How to transform AVRO (IDL) data to multiple PARQUET files
summary
A library that converts Avro IDL events into multiple Parquet extension tables and fact/dimension (star/snowflake) schemas, automating flattening of deeply nested records so consumers avoid manual Spark work. It emits Delta Lake tables with unique version hashes, runs as lightweight .NET/Java containers deployable to Kubernetes (far cheaper than 24/7 Spark clusters), supports querying a trillion rows in ~6–7s on Delta Lake (architecture-dependent), is open source under LGPL, and awaits formal CO2/storage savings verification.
AI-written summary · about
chapters (9)
- 0:00Avro and Parquet intro
- 5:14Why Pandora chose Kafka
- 14:23Avro to Parquet challenges
- 23:22Designing extension tables
- 31:08Automating with Delta Lake
- 37:01Sustainability impact
- 42:00Open sourcing library
- 49:12Demo and implementation
- 57:34Q&A
related talks
39mDelta Lake: Diving into Data Lakes Without the Downsides
58mBuild reliable data lakes with Delta Lake & Databricks
31mMapping FHIR data to parquet for analytics and machine learning
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
30mReconciling Kafka, SQL and Parquet
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
30mDelta Lake and Delta Sharing
46mEfficient Data formats for Analytics with Parquet and Arrow