JOINs Are Slowing You Down: Why Spark and Trino Won’t Replace Your DWH
summary
Examines why JOIN-heavy workloads can be slow and why query engines like Apache Spark and Trino may not be suitable substitutes for a dedicated data warehouse. Explains how query execution engines work, what execution-engine characteristics (beyond RPS and cluster fault-tolerance) affect query behavior and performance, and how those factors influence database selection and tuning.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:12Join problem and combinatorics
- 3:18Cost-based optimizer
- 5:25Top-down vs bottom-up
- 8:59Implementation phase
- 11:05Join topology and star schemas
- 12:10Bottom-up algorithms
- 15:23Evolution of bottom-up
- 17:08Foreign keys and data vaults
- 18:29Statistics in optimization
- 23:04Trino vs Spark vs Greenplum
- 29:21IDB architecture
- 31:50Summary
- 34:43Q&A
related talks
27mHarnessing the Power of Dual Query Engines.
15mOptimizing Query Performance in Big Data Systems
01h 07mAn Overview of the Starburst Trino Query Optimizer
50mChallenges that everyone struggles while productionizing Apache Spark workloads
39mStorage-Partitioned Join for Apache Spark
28mAn Apache Spark query's journey through the stack
41mOptimising Apache Spark and SQL for Improved Performance
41mParallel and distributed big joins in H2O