Challenges of Machine Learning at Scale
summary
A practical design for an ad-click pipeline that makes 100,000 queries/sec, handles gigabytes/sec of logs and trains over billions of features uses map-reduce style feature extraction, join of impressions to delayed click labels, sharded online learners that average updates, periodic checkpoints (example: every few hours or eight hours) and serving by model shards with background hot-swap loading. Failure modes include checkpoint rollback on crashes, spam-labeled clicks requiring backfills, and lookup/lookup-miss latency from external databases; the sharding-plus-averaging approach enables parallel updates but requires careful checkpoint and freshness tuning.
AI-written summary · about
chapters (5)
- 0:00Start
- 2:21Overview
- 10:08Example System Rec
- 23:47Sharding the learning
- 38:07Question & Answer a Scaling Machine Lea
related talks
27mLarge Scale Response Prediction to Serve Ads on LinkedIn
37mAutomated Machine Learning: Building a Pipeline
41mFrom 5 PB of Daily Logs to Thousands of ML Models #bbuzz
41mInfrastructure for ML Applications
38mBuilding robust machine learning systems
18mDesigning a scalable ML Ops Pipeline: Insights and best practices
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
33mMachine Learning to moderate ads