The Laws of ML at Scale
summary
Three concrete laws: scaling triggers a vicious cycle where ad hoc infra requests (Spark, GPUs) balloon cost and never finish; early trade-offs compound into brittle dependency webs (Spotify ends up with 25,000 workflows, 1,000 repos across 300 teams; Lyft used ~600 workflows with an 8–9 person team); and rising complexity produces negative work that kills value (migrations and reproducibility cost effort). A solution shown is Flyte: versioned, language-agnostic tasks, Spark↔Pandas via Arrow/Parquet, caching, declarative infra (GPU/Spark/databricks) and a one-month vendor migration with no code changes.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:52Laws of scale in ML
- 6:22Vicious cycle of ML infra
- 8:48Engineering trade-offs at scale
- 12:16Reducing negative work
- 14:42Adaptable systems for value
- 17:33Summary
related talks
41mInfrastructure for ML Applications
28mUnravelling Hidden Technical Debt in ML: A Pythonic Approach to Robust Systems
34mScaling ML in the Traditional Enterprise
26mManaging ML Models @ Scale
13mMLOps at Scale
29mScaling to Millions of ML Models to Solve the Problems of SRE and Security
38mReal Time ML Pipelines in Multi-Tenant Environments
39mMachine Learning with Scala on Spark