Productionizing Big Data - Stories from the Trenches
summary
Company runs ~60 team workloads and hundreds of jobs on Scala with Spark, Flink, HDFS/Parquet, Impala, S3 and GCS and migrates to Databricks. Real incidents: a 13‑month user profile gap is fixed by incremental one‑month reloads; a week of missing data is caused by renamed columns and handled with regex-based renames to accept both schemas; slow Spark job doubling from 3–4h to ~6h is solved by hash-based seed partitioning (0–99 buckets); event timing issues are fixed by Databricks autoloader bronze tables per event type. Each fix reduces reprocessing and stabilized downstream reports.
AI-written summary · about
chapters (9)
- 0:00Productionizing big data pipelines
- 4:23Unlucky query and data loss
- 9:42Missing data and column renaming
- 13:57Spark job partitioning issues
- 18:55Data mesh evolution
- 27:01Event-based pipeline partitioning
- 29:31Pipeline modernization and cleanup
- 33:02Key lessons for production data
- 39:21Conclusion
related talks
57mScaling Databricks to Run Data & AI Workloads on Millions of VMs
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
01mDatabricks Customers
18mDatabricks drives predictive maintenance, grid optimization with analytics
49mBuilding a Data-Driven Bank
02mDatabricks at GoDataFest 2019
26mRe-architecting the Data Platform with Spark on Kubernetes