How ML Breaks: A Decade of Outages for One Large ML Pipeline
summary
Google analyzes 96 post-mortems from a 15-year continuous pipeline running thousands of concurrent models with about 100 billion parameters, finds failures sorted into 19 failure categories, and reports that the majority of outages are not ML-specific but stem from distributed-system issues (or mundane infra like cache invalidation, joins, dependency moves, CPU faults). The team builds a taxonomy, counts frequencies, and recommends pragmatic fixes: throughput and age monitoring, strict versioning, capacity tracking, canarying, and better distributed-systems engineering.
AI-written summary · about
chapters (6)
related talks
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
46mPanel: OpML
27m10 Things That Can Go Wrong with ML Projects
10mML-based Log Analysis for Faster Debugging
25mBuilding Scalable MLOps Pipelines That Actually Work
28mMastering MLOps: Building Future Ready ML Systems
33mBuilding Resilient (ML) Pipelines for MLOps
18mDesigning a scalable ML Ops Pipeline: Insights and best practices