SwiftLake: Bringing Apache Iceberg and DuckDB Together for a Smarter Data Management
summary
SwiftLake presents a concrete bridge from RDBMS to S3 by packaging Apache Iceberg and DuckDB as a single-node Java library that supports full write paths (INSERT/UPDATE/DELETE and MERGE) not just reads. It enforces schema evolution, offers time travel and snapshot/branch queries, implements SCD type two and type one merge semantics, and does local caching plus hidden partitioning and Parquet pushdowns, all while exposing JDBC/MyBatis-compatible APIs; the library is open source on GitHub.
AI-written summary · about
chapters (9)
- 0:00Introduction to RCCM and challenges
- 5:26Problem with RDBMS and S3 architectures
- 7:53Introducing SwiftLake solution
- 10:01SwiftLake setup and basic operations
- 13:57Advanced operations merge and SCD
- 20:33Performance features and caching
- 22:41Time travel and snapshot capabilities
- 24:27Comparison with other tools
- 25:33Key takeaways and future vision
related talks
40mDuckDB Meets Data Lakes
26mBuilding Data Lake Using DuckDB
47mWhy DuckDB could be the engine for your Data Lake?
25mDelta Lake has competition
28mAnalyze data easily with duckdb - and the implications on data architectures
11mBreaking Data Silos: Enterprise Data Management
22mThe Who, What, and Why of Data Lake Table Formats
40mDuckDB beyond the notebook