DuckLake

DuckLake is an integrated data lake and catalog format designed to provide advanced data lake capabilities with reduced complexity. It operates as a lakehouse format built on SQL, using Parquet files for storage and a SQL database for the catalog. DuckLake is positioned for users seeking a lightweight solution for managing large datasets, supporting both local and cloud deployments without vendor lock-in.

The platform offers a range of features including ACID-compliant transactions for multi-table operations, lightweight snapshots, time travel queries, schema evolution, and data partitioning. Snapshots can be created in unlimited numbers without requiring frequent compaction, and the system supports concurrent access with transactional guarantees. Performance is enhanced through the use of statistics for filter pushdown, enabling efficient queries even on substantial datasets.

DuckLake's architecture separates the catalog and storage layers: the catalog is maintained in an ACID-compliant SQL database, while data is stored in Parquet files that can reside either on local disk or in object storage. The tool is compatible with multiple SQL databases for the catalog, including PostgreSQL, SQLite, and DuckDB, and can be accessed from various clients. DuckDB provides first-class support for DuckLake and enables a "multiplayer DuckDB" experience, allowing multiple DuckDB instances to read and write to the same dataset concurrently, a feature not available in standard DuckDB.

Installation and setup involve using the DuckLake DuckDB extension, with support for attaching catalogs via PostgreSQL, SQLite, or DuckDB itself. 0, which was released with guaranteed backward compatibility. The specification and the DuckLake DuckDB extension are distributed under the MIT license, and the project is developed by the DuckDB Foundation based in Amsterdam. This open, standalone format is intended for users who need robust data lake and catalog management without the typical complexity found in traditional lakehouse solutions.