Yes, you can develop against the Databricks API on your own machine: MiniLake 1.9.0 emulates it locally with real SQL on DuckDB, real Delta Lake files, and real Spark jobs. We tested it and its complete suite passed 406 out of 406 tests. It’s free, MIT-licensed, and doesn’t require an account, but the PyPI version has a bug worth knowing about before you install it.
yoDEV Tested · October 29, 2026
We tested MiniLake 1.9.0, both the PyPI package and theghcr.io/dmux/minilake:1.9.0image, withdatabricks-sdk0.158.0, and ran the project’s complete test suite at tagv1.9.0in a cloud Linux environment. The results we report as ours come from that test.
MiniLake is an open-source project by Rafael Sales, first published to PyPI in July 2026; version 1.9.0 arrived there on October 25. We installed it, pointed the official Databricks SDK for Python at it, ran the entire project test suite, and looked for places where it diverges from what code written for Databricks expects. Here’s what we found.
What is Databricks and what is MiniLake?
Databricks is the cloud data and analytics platform built around Apache Spark and Delta Lake. Developing for it typically requires a workspace, a token, and compute that you pay for while you iterate. MiniLake is a local emulator of the Databricks REST API: a single process on port 8000 that speaks the same protocol as a workspace, so databricks-sdk, the Databricks CLI, the Terraform provider, and Asset Bundles can connect to it by only changing the host.
What sets it apart from a mock server is what’s behind the API:
- SQL runs on DuckDB. Each Unity Catalog is its own DuckDB database, so the
catalog.schema.tableaddressing is native. - EXTERNAL Delta tables are real files: Parquet plus a
_delta_log/on disk. Writes go through a generated Spark job; reads go through DuckDB’sdelta_scan(). - Jobs run real
spark-submitin sibling Docker containers, with DAG scheduling that respectsdepends_onandrun_if. - What it doesn’t emulate responds with
501 NOT_IMPLEMENTEDinstead of pretending it worked.
The project declares over 30 emulated API groups and an MCP server with 67 tools for AI agents. These numbers are from the project itself.
What is Delta Lake?
Delta Lake is the open table format that Databricks uses: Parquet files accompanied by a transaction log (_delta_log/) that adds ACID transactions, versioning, and history. That MiniLake writes real Delta means the files it generates locally have the same format as those in a Delta table in production.
Is MiniLake free?
Yes. It’s MIT-licensed, doesn’t require a Databricks account, API key, or registration, and runs entirely on your machine. MiniLake is not affiliated with Databricks nor endorsed by it.
MiniLake or Databricks Free Edition?
They don’t compete: they solve different parts of the cycle. Databricks Free Edition is the real platform, free but with quotas; MiniLake is a local emulator without quotas. According to Databricks documentation consulted on October 29, 2026 (updated October 18):
| MiniLake | Databricks Free Edition | |
|---|---|---|
| Where it runs | On your machine or in your CI | On Databricks cloud |
| Account | Doesn’t need one | Databricks account |
| Commercial use | Yes (MIT license) | No, non-commercial only |
| SQL dialect | DuckDB | Databricks Spark SQL |
| Compute | Your hardware; Spark in Docker containers | Serverless only; one 2X-Small SQL warehouse |
| Limits | No quotas; single process | Daily quotas: exceed them and compute shuts down until they reset |
| Offline | Yes, with Docker image | No |
In practice they complement each other: MiniLake for the fast cycle and CI, Free Edition to test against the real platform what the emulator can’t guarantee, like the SQL dialect.
How do I install MiniLake?
The fastest way is Docker, and for version 1.9.0 it’s also what we recommend:
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
Then verify it’s running:
curl http://localhost:8000/_minilake/health
The image comes with the DuckDB delta extension and the Delta and Unity Catalog jars built in, so it doesn’t download anything on startup. We verified this: started with --network none, the 1.9.0 image booted healthy with no network access.
The integrated SQL workspace, an Athena-style editor with data catalog, saved queries, and history, is at http://localhost:8000/ui/.
What about pip?
It also works with pip:
pip install minilake
minilake --port 8000
But as of the time of publishing this note (October 29, 2026), the PyPI version 1.9.0 has a bug: with the default data directory, all writes to DBFS and the Files API fail with Invalid path. The path check compares an absolute path against the relative directory ./data it uses by default, so it always rejects it. Docker isn’t affected because the image defines an absolute data directory.
The fix is to start MiniLake with an absolute data directory:
MINILAKE_DATA_DIR="$PWD/data" minilake --port 8000
We reproduced the bug and the fix with the PyPI package. The fix is already integrated into the main branch of the repository, where the same tests pass, but at the time of writing it hadn’t been published to PyPI yet. Also, pip installation downloads the delta extension the first time it’s needed, so the guarantee of offline operation applies to the Docker image, not pip.
One more detail if you’re already using DuckDB 2.0: MiniLake 1.9.0 requires a DuckDB version earlier than 2.0 (duckdb>=1.0,<2), so it installs a 1.x version. If you’ve already migrated your projects to DuckDB 2.0, install MiniLake in its own virtual environment or use Docker. Our guide to migrating to DuckDB 2.0 explains what changes between the two.
How do I connect the Databricks SDK for Python to MiniLake?
Change the host. MiniLake accepts any token:
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType, DataSourceFormat
w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
name="orders", catalog_name="sales", schema_name="store",
table_type=TableType.MANAGED,
data_source_format=DataSourceFormat.DELTA,
storage_location="/data/sales/store/orders",
columns=[
ColumnInfo(name="id", type_text="BIGINT"),
ColumnInfo(name="customer", type_text="STRING"),
ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
],
)
wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
warehouse_id=wh.id,
statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)
result = w.statement_execution.execute_statement(
warehouse_id=wh.id,
statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array)
There’s one difference from the example on MiniLake’s website: data_source_format uses the DataSourceFormat.DELTA enum. The website passes the string "DELTA", which the current SDK (we tested databricks-sdk 0.158.0) rejects with AttributeError: 'str' object has no attribute 'value'. With the enum, the insert and query work correctly in about 50 milliseconds.
For CI, POST /_minilake/reset gives each test a clean workspace without restarting the container, and MINILAKE_PERSIST=1 preserves state between restarts.
What did we test and what results did we get?
We ran three tests on October 29, 2026:
- A quick test of the PyPI package with the official Python SDK.
- The project’s complete test suite at tag
v1.9.0, with the project’s own Docker Compose command. The suite uses the real Databricks SDK, the Databricks CLI, and the Terraform provider. - The published Docker image
ghcr.io/dmux/minilake:1.9.0, with and without network access.
| Test | Result |
|---|---|
Complete test suite, v1.9.0 |
406 passed, 0 failed, 0 skipped, 3 min 56 s |
| Real Spark jobs in sibling containers | Passes |
| INSERT, UPDATE, and DELETE on Delta over real files | Passes |
Spark reading cat.sch.tbl via Unity Catalog protocol |
Passes |
| Terraform apply, plan, and destroy | Passes |
Docker image with --network none |
Boots, healthy |
| DBFS in PyPI 1.9.0 with default directory | Fails (Invalid path) |
| Website example with current SDK | Fails, fixed in one line |
The project claims over 378 tests using the real SDK; the suite we ran comprises 406.
Does MiniLake use Spark SQL or DuckDB?
DuckDB, and it’s the most important thing you need to understand before adopting it. SQL statements execute in DuckDB’s dialect, not Spark SQL. The project states this clearly, and our test showed how quickly you run into it:
| Function | Dialect | Result in MiniLake |
|---|---|---|
date_format(current_date(), 'yyyy-MM-dd') |
Spark SQL | Error: function doesn’t exist |
nvl(NULL, 1) |
Spark SQL | Error: function doesn’t exist |
collect_list(customer) |
Spark SQL | Error: function doesn’t exist |
strftime(current_date, '%Y-%m-%d') |
DuckDB | Works |
That’s why SQL written for a Databricks SQL warehouse often needs translation. The project includes a dialect equivalence table in its documentation, and for agents, a specific resource for the same reason. Code that goes through Spark (jobs, spark.table()) is another story: that does run on real Spark. If you’re looking for the DuckDB vs Spark comparison, this is the practical boundary: DuckDB for SQL statements, Spark for jobs.
We also observed that result manifests describe columns in DuckDB vocabulary: a column declared STRING comes back with type_text: VARCHAR. If your code inspects result schema, test that path. We didn’t compare it with a real workspace.
What doesn’t MiniLake do?
By design, and according to its own documentation:
- No authentication. Any token works and there’s a single user. Never expose it beyond localhost.
- No access control. The permissions API is a real CRUD that always allows everything, so a passing test says nothing about permissions.
- No Spark in clusters. Clusters are a state machine; real Spark runs via Jobs.
- Single process, no high availability. DuckDB supports a single writer, so concurrent load is serialized.
- MANAGED tables are invisible to Spark. Use EXTERNAL Delta tables when Spark needs the data.
- Not implemented: Delta Live Tables, Model Registry, Vector Search, Lakeview, and Repos. We called the pipelines API and got a clear
NotImplementederror.
Can AI agents like Claude Code use MiniLake?
Yes. MiniLake includes an MCP server, disabled by default:
MINILAKE_MCP=1 minilake --port 8000
claude mcp add --transport http minilake http://localhost:8000/mcp
The second line registers it with Claude Code. The benefit is both security and convenience: an agent can create catalogs, load tables with data, and run Spark jobs against a lakehouse it can’t break and that costs nothing when it makes mistakes. The MCP tests in the suite, including running a real Spark script through the agent’s tools, passed in our test.
When should you use MiniLake?
It fits when what you want to speed up is the API cycle: Python code based on databricks-sdk, Terraform modules, Asset Bundles, Spark jobs, CI pipelines that today need a workspace, and agents you want to give freedom without a cloud bill.
It fits worse when your work is mostly SQL written for Databricks SQL warehouses (because of the dialect difference), or when you need to validate permissions, Delta Live Tables, or anything else from the not-implemented list. And it’s a development tool, not a place to run anything shared.
Our verdict: for an emulator of such a large platform, honesty is the main feature. What it emulates ran for real in our tests, and what it doesn’t emulate tells you so loudly. Install it with Docker for now and keep the dialect difference in mind from your first query.
What about you? Where do you test your Databricks code today before taking it to production?
- Spark jobs and notebooks
- Deployments with Terraform or Asset Bundles
- Tests in CI
- I prefer another option (tell us which)
Comment below or, if this article reached you by email, reply directly to the email: your reply gets published here.
Related: DuckDB 2.0: how to migrate from 1.4 and 1.5 without breaking your database or your Python code