Key lessons from building Data and AI systems over the last 26 years…
summary
26 years of systems work distills into concrete claims: built core pieces of SQL Server crown jewel and re-engineered OLTP for cloud; moved to Databricks and shipped Spark SQL extensions made in Belgrade. Highlights include a single table’s metadata ballooning to ~1 TB, solving it with a metadata catalog centralizes lake approach; diagnosing a production failure as Noisy Neighbor overbooking; only ~15 engineers built a leading open model; about 20–30% of orgs deploy AI to production, and short cycle times decide success.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 3:42Database systems foundation
- 7:13Data and hardware demands
- 10:04Metadata challenges in analytics
- 11:51Metadata and engine coexistence
- 12:53Performance engineering complexity
- 14:21AI hype versus reality
- 17:50Data science team lessons
- 21:45Importance of system tails
- 23:34Deploying AI systems
- 26:29Business and personal stakes
- 29:20Prototyping and manual work
- 30:26Open source motivations
- 32:57Power of incentives
- 35:31Long cycle times risks
- 36:54Future of data and AI
- 37:57Q&A on time cycles
- 40:49Microsoft to Databricks
- 43:18Closing remarks
related talks
13mCombining algorithms and quality data to build better AI platforms
41mData Engineers: Respect the Resiliency of SQL
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
01mDatabricks Customers
53mKeynote: From RAP to Snowflake - A Look at 50 Years of SQL DB Scalability
58mTales of Data Architecture Evolution
27mEnterprise AI at Scale: Success Stories and Insights
35mBuilding Data-Intensive Teams