Unveil the Magic Without Hoodini: Transform Your Machine Learning Pipelines with Apa...
summary
Apache Hudi provides upserts and record-level updates plus a timeline and metadata table so pipelines only process deltas instead of full rewrites; example uses Debezium -> Kafka -> Hudi bronze -> Hudi silver upserts -> incremental joins into gold, then models. At production scale (ByteDance at exabyte) Hudi cuts downstream refreshes from days to minutes. Recent 1.0 beta adds CDC mode with before/after images and non-blocking concurrency control for high-volume streaming writes.
AI-written summary · about
chapters (13)
- 0:00Apache Houdini intro
- 1:07Medallion architecture
- 3:56Medallion bottlenecks
- 5:39Incremental processing in Houdini
- 7:25Houdini as lakehouse
- 9:50Hoodie table structure
- 12:43Hoodie at scale
- 13:23Hoodie Streamer pipeline
- 15:08Record mutations in Hoodie
- 18:24CDC mode in Hoodie
- 19:06ML pipeline case study
- 21:36Hoodie roadmap
- 23:20Q&A
related talks
19mDemystifying Modern Data Pipeline Architecture
44mImprove your App UX with one model processing insights from batch and streams
11mBig Data Moonshots and Ground Control
29mData Acquisition: Overcoming Challenges
25mBuild open format data lakehouse
31mHow to quickly build Data Pipelines for Data Scientists
22mThe Who, What, and Why of Data Lake Table Formats
11mBreaking Data Silos: Enterprise Data Management