Data Engineers: Respect the Resiliency of SQL
summary
Claim: SQL still powers real systems and now runs ML and streaming pipelines end to end. Examples: BigQuery lets users create model with hyperparameters train evaluate and call ml.predict to score new rows and schedule jobs; KSQLDB scripts consume Kafka events compute geodistance and emit to a new topic in SQL; dbt reuses models and CTEs and Iceberg time travel run as SQL. The talk shows these things built and working, and highlights WITH CTEs and window functions.
AI-written summary · about
chapters (13)
- 0:00SQL’s resiliency and evolution
- 2:15Tardigrades prove SQL indestructible
- 4:42Three principles behind SQL design
- 7:55SQL beyond original purpose
- 9:20Window functions and CTEs
- 13:16SQL’s Darwinian evolution
- 16:45Google’s Pipe SQL syntax
- 17:27SQL in Kafka and real-time
- 20:38DBT and SQL transformations
- 22:47Open table formats and storage
- 24:15ML in SQL with BigQuery
- 28:07SQL as universal language
- 29:35Q&A
related talks
02h 42mSQL (The Return): Demystifying Preconceptions and Using the DB Effectively
30mThe Future of Data is Words
44mNonrelational SQL
40mOne SQL to Rule Them All
43mKey lessons from building Data and AI systems over the last 26 years…
12mPower a Better Future
54mKeynote: SQL Edge to Cloud
34mThe State of SQL-Based Observability