Data Demons: Unleashing Chaos or Driving Performance?
summary
Twenty years of hands‑on incidents distilled into concrete fixes: turning a $50,000 IoT data dump that causes the vampire of slow queries into a two‑table hot/cold design, using Airflow automates lifecycle to move month‑old rows to S3 and query history via Trino/Starburst; cutting BigQuery scan costs by repartitioning on region/status and using built‑in time functions; fixing onboarding by swapping to a storage/compute separated warehouse with automated DDL generation; reordering Spark stages to stop daily failures and save ~30% cost while eliminating a ~40% onboarding churn.
AI-written summary · about
chapters (6)
- 0:00Introduction to data demons
- 1:56Vampire of slow queries
- 9:35Costly data channel demons
- 15:56Onboarding data inconsistency
- 19:56Spark cluster failure
- 23:56Final advice and summary
related talks
36mMigrating Our Data Warehouse to the Cloud
58mTales of Data Architecture Evolution
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
40mProductionizing Big Data - Stories from the Trenches
30mData Engineering building your BI infrastructure from scratch
31mFrom Raw Data to Real Impact: Lessons from Real-World Transformations
40mFinOps Examples using Google Cloud BigQuery by Aliz.ai
37mData Swamp to Data Swimming Pool: Cleaning up the Mess