Keynote: From RAP to Snowflake - A Look at 50 Years of SQL DB Scalability

on YouTube ↗53m

" summary

A 50‑year retrospective traces early database machines (RAP, Chasm, Raries) as academic prototypes that mostly fail, the first parallel systems (Direct runs but hits n^2 messaging), and Gamma which applies Grace hash join to parallel shuffles and achieves real speedup (by 1986 joins of ~1,000,000×100,000 rows). Teradata becomes a successful appliance (customers storing ~10 PB on 256 nodes by 2011), the shared‑nothing era commoditizes scale, and the cloud brings Snowflake separation of storage and compute, Redshift lift‑and‑shift, Databricks Photon (3×–10×) and adaptive query execution.

AI-written summary · about

" chapters (29)

1. 0:00Introduction and speaker background
2. 1:58Retirement and consulting work
3. 3:20Three eras of SQL database scalability
4. 4:42Relational model debate and early systems
5. 6:05Database machines and custom hardware
6. 9:16Failure of database machines and Direct project
7. 11:25Wisconsin benchmark and early performance results
8. 14:13First appliances: Britton-Lee and Teradata
9. 15:39Shared nothing architecture explained
10. 17:27Partitioning strategies and skew handling
11. 19:31Teradata success and custom hardware details
12. 21:42Roll your own era begins
13. 22:25Evolution of join algorithms
14. 23:53Grace hash join and its impact
15. 26:46Parallel joins in shared nothing systems
16. 29:15Gamma project and hardware setup
17. 31:21Mike Stonebraker and shared nothing revolution
18. 33:11Critique of MapReduce and big data paradigms
19. 34:59Cloud computing paradigm shift
20. 36:24Pre-cloud cost and complexity comparison
21. 37:50Cloud advantages and time to insight
22. 39:18Cloud database systems overview
23. 40:23Amazon Redshift architecture and features
24. 42:53Snowflake architecture and innovation
25. 46:08Apache Spark as a game changer
26. 47:51Spark performance improvements with Photon
27. 48:34Query optimization challenges and solutions
28. 50:23Adaptive query execution and future trends
29. 51:2950 years of SQL scalability and closing remarks

" related talks

• 13mLightning Talk: Highlights of the Snowflake Elastic Data WarehouseTorsten Grabs · FoundationDB Summit 2018
• 50mA Short Summary of the Last Decades of Data ManagementHannes Mühleisen · GOTO Amsterdam 2024
• 37mKEYNOTE: Data Architecture Turned Upside DownHannes Mühleisen · PyData Amsterdam 2025
• 58mTales of Data Architecture EvolutionJosef Goldstein · NDC Oslo 2023
• 27mScaling for Intelligence: Distributed SQL Databases in the Age of AIWei Hu · DevWorld 2026
• 58mUnderstanding Big Data for Software EngineersDavid Ostrovsky · NDC Oslo 2023
• 38mKeynote | Data Warehouse, Data Lakes, What's Next?Xiaowei Jiang · Flink Forward Global Virtual 2020
• 43mKey lessons from building Data and AI systems over the last 26 years…Dragan Tomic · DSC Europe 24