Making Iceberg walk and talk like Postgres
summary
Project brings Iceberg into Postgres as a native-backed table format, reimplemented Iceberg metadata in C and kept metadata cached in Postgres, delegates heavy scans to DuckDB running in a local PG Duck server sidecar to avoid per-connection overhead, and supports atomic transactions that update Postgres and Iceberg metadata together. Builds a workflow that flushes hot Postgres inserts (measured at 100–200k inserts/sec) into Iceberg every 10 seconds, plus a REST catalog sync and tools for incremental shoveling.
AI-written summary · about
related talks
01h 12mHands-on with Apache Iceberg
27mBuilding a PostgreSQL data warehouse
25mpg_lake: Postgres as a lakehouse
41mTip of the Iceberg
26mThis Impala not only reads, but modifies and optimizes Iceberg tables
28mBridging Python and Apache Iceberg: The Power of PyIceberg
04h 15mTopical Edition on Databases - Track 2
37mApache Iceberg: An Architectural Look Under the Covers