Big Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
summary
Big Data serving claims real‑time selection and inference over 100 billion content items with latencies under 100 milliseconds, backing 300,000 requests/sec in production. The platform moves execution to data, runs TensorFlow and XGBoost models in a C++ content layer, supports real‑time writes, CRUSH-based automatic rebalancing, approximate nearest neighbor with filters, and an admin cluster that deploys application packages online. Benchmarks show local model evaluation scales; shipping raw tensors across the network does not.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:20Big Data Serving
- 3:30Latency
- 4:36Advantages
- 10:07Why arent everybody already doing this
- 11:56What is RESPA
- 13:34How RESPA is used
- 14:58Largescale search
- 18:26Recommendation in recommendation
- 21:02Use case for inspiration
- 22:46Analytics vs Big Data Serving
- 25:07Recap
- 27:34Architecture
- 31:06Scaling Benchmark
- 32:25Technical Details
- 34:12Data Distribution
- 35:38Inference Engine
- 38:17Release Schedule
- 39:04Conclusion
related talks
19mBuild Real time, AI powered Apps
27mDataEngBytes 2023 - SYD-T1-10 - Kanishka Mohaia
40mScalable machine-learned model serving
24mLarge Scale On-Demand Low-Latency Near Real-Time Predictions
37mBig Data Analytics, Machine Learning, Real Time Processing
18mCluster Serving: Distributed Model Inference using Big Data Streaming
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
13mScaling AI Systems for High-Volume Transaction Processing