Beyond One Model: Scaling, Orchestrating & Monitoring
summary
Shows a deployed anomaly-detection fleet that moves from a single model to per-shuttle models, claims scaling to hundreds (and a path to thousands) by running parallel tasks per site, hourly aggregations and daily inference, with training still manually triggered. Implementation uses Databricks with Spark declarative pipelines for ETL, legflow jobs (orchestration) for complex workflows, GitHub-based CI/CD that builds wheels into an artifactory and deploys via Crossplane and Terraform, plus a dashboard and job-health monitoring for alerts.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 2:34Model building to pipelines
- 4:45Many models
- 7:54Predictive maintenance at Fondelanda
- 9:22Storage failures
- 11:54Scaling hundreds of models
- 14:47Scaling pipeline requirements
- 15:50Modular pipelines attempt
- 18:19Data isolation and scalability
- 19:02Fault-tolerant pipelines
- 20:29Inference orchestration
- 22:14DevOps pipeline deployment
- 24:25Monitoring models and jobs
- 26:10Future automation goals
- 27:16Lessons learned
- 28:19Q&A
related talks
29mScaling to Millions of ML Models to Solve the Problems of SRE and Security
27mDeveloping and Deploying Models at Scale — without Spark!
33mDealing with the versioning of production-ready models
33mDeploying and Scaling Spark ML and Tensorflow AI Models
44mBuilding Multi-Tenant Compute Systems in the Enterprise
21mMastering the DevOps Machine Learning Pipeline
28mProductionizing Machine Learning Models at Scale with Kubernetes
49mOrchestrating data and ML workflows with Apache Airflow