ML Latency No More- Useful Patterns to Reduce ML Prediction Latency to Sub X ms
summary
Argues that latency is a business constraint not a nicety: treat ~100 ms as the cutoff (examples use <10 ms timeouts), measure optimizing vs satisficing metrics, and move heavy work offline into batch jobs and a low‑read‑latency feature store. Describes an ML Gateway, precomputed prediction stores with top‑N cache + fallback live predict, and vector search with Annoy/ScaNN. Claims these patterns cut serving latency at the cost of engineering and storage tradeoffs.
AI-written summary · about
chapters (17)
- 0:00ML latency challenges
- 1:47Why latency matters
- 3:53ML abstraction layers
- 7:05Online prediction pain
- 8:28Split construction and serving
- 10:13Optimizing vs satisficing
- 12:17Distributed I/O vs local latency
- 13:00Model input features
- 15:50Dynamic real-time features
- 17:56Pre-computation strategies
- 18:59Entity cardinality handling
- 21:56Cost-latency trade-offs
- 23:22Exact vs approximate matching
- 25:51Caching predictions
- 26:55Primitive whole systems
- 28:46Offline to online transition
- 30:54Question and answer session begins
related talks
25mMachine Learning is Going Real-Time
41mChallenges in incorporating ML models
19mOn Modular Learning of Distributed Systems for Predicting End-to-End Latency
11mLitePred: Transferable and Scalable Latency Prediction for Hardware-Aware Neural...
41mInfrastructure for ML Applications
52mMachine Learning in IoT Applications
31mLow-Latency Solutions for Resource-Constrained Environments
14mPerformance Optimization in High-Volume Transaction Systems