LightningTalk: MultiRay: An Accelerated Embedding Service for Content...
summary
MultiRay is an embedding service that computes and caches encoder-based embeddings (XLM-R and successors) for text, image and multimodal posts, exposing self-describing, quantized embeddings and simple RPCs. It serves TextRay, ImageRay, PostRay across 125 use cases at 800B requests daily and ~20M QPS, uses multi-level caching (memcached + flash) with cost- and reuse-aware replacement, batches requests for accelerator efficiency, and keeps <10% of queries hit accelerators by trading storage for recompute avoidance.
AI-written summary · about
chapters (11)
- 0:00MultiRay introduction
- 1:27LLM deployment challenges
- 4:24Embeddings and caching solution
- 6:09Multi-level caching algorithm
- 8:18Accelerators and batching role
- 9:25Amortizing accelerator costs
- 9:45System architecture overview
- 11:11Centralized embedding benefits
- 11:56MultiRay evolution
- 12:37Performance metrics impact
- 13:18Q&A
related talks
22mRay, a Unified Distributed Framework for the Modern AI Stack
31mTransforming Multimodal Data Management with LanceDB-Ray
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
13mRay and Anyscale: An Optimization Journey
28mScaling Multimodal Data and Reinforcement Learning with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
44mPanel: ML Platform on Ray
26mA Growing Ecosystem of Scalable ML Libraries on Ray