Building highload ML powered service
summary
Asserts Python backends can run high‑load ML in production and demonstrates it with a transaction‑enrichment pipeline handling millions of samples, supporting six languages and an SLO of B95 latency <200 ms. Implements model distillation and shared base layers, uses Triton inference server with dynamic batching and autoscaling, moves latency‑critical code by migrating hot paths to Rust, adds smart caching with locks, blue‑green deploys, and large‑scale sentence clustering for labeling; it works at scale.
AI-written summary · about
chapters (6)
- 0:00Highload ML services intro
- 3:35Scaling ML development
- 10:19Production throughput for ML
- 25:29Handling new data patterns
- 31:25Summary
- 32:07Q&A on model validation
related talks
14mPerformance Optimization in High-Volume Transaction Systems
41mInfrastructure for ML Applications
33mBuilding Resilient (ML) Pipelines for MLOps
29mMachine learning in production
24mBuilding an ML Application Platform from the Ground Up
51mSpeeding up deployment of machine learning models
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
32mLiveMLP: ML Platform for Assisting Contact Center Agents in Real-Time