From Storage Formats to Open Governance: The Evolution to Apache Polaris
summary
Snowflake engineers build and contribute Apache Polaris, an Iceberg REST catalog that shifts concurrency, conflict resolution and snapshot creation from clients into a server, enabling roll back compaction deconfliction (avoids wasted 8‑hour jobs) and a multi‑table commit API. Polaris adds role‑based access control, credential vending scoped to S3 prefixes, event/audit lineage, server‑side planning and a “finalize commit” flow for streaming writers. It handles bloated metadata (customers hit ~200 MB JSON files), federates other catalogs, and prototypes fine‑grained row and column policies with trusted‑engine enforcement.
AI-written summary · about
chapters (11)
- 0:00Apache Polaris and Iceberg intro
- 1:03Iceberg catalog and concurrency
- 9:46Storage to relational catalogs
- 16:52Metadata responsibility shift
- 23:53Iceberg REST catalog benefits
- 27:45Polaris conflict resolution
- 31:16Polaris multi-table transactions
- 40:10Polaris governance layer
- 48:15Polaris RBAC auth
- 56:18Polaris policy store
- 1:06:33Community and Q&A
related talks
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
34mApache Iceberg.
38mHerding Cat(alogue)s: The Apache Iceberg™ Catalog Landscape
50mDesigning a scalable data platform based on Apache Spark/Iceberg
39mMetadata as a Control Plane: Inside the Apache Iceberg REST Catalog
37mApache Iceberg: An Architectural Look Under the Covers
33mStreamlining Metadata Management with PostgreSQL and Apache Gravitino
42mPOLARDB architecture