Intelligent Data Classification with Ray and vLLM at Apple
summary
Apple shows a working pipeline that fine tunes LLMs with Ray and vLLM and Ray for low latency online and high throughput batch metadata generation, uses quantization to cut memory and cost and Ray autoscaling (min 1 max 10, target ~5 reqs) to serve many tables. An offline batch job + feedback store produces training data from roughly 10% of datasets to fine tune models, and a homegrown lineage system feeds prompts so outputs steadily improve with human review and retraining.
AI-written summary · about
chapters (6)
- 0:00Introduction to data classification
- 3:12Challenges in manual annotation
- 8:51Limitations of large language models
- 12:02Fine-tuning for accuracy
- 16:46Inference with vLLM and Ray
- 21:24Key takeaways and future impact
related talks
28mServing LLMs for Agents at Scale with Ray + vLLM
28mScaling Multimodal Data and Reinforcement Learning with Ray
21mEnd-to-End AutoML with Ludwig on Ray
11mvLLM & Ray Updates
12mKeynote: How LLMs Have Changed the Way Organizations Do AI
23mAn Open Source Post-Training Stack: Kubernetes + Ray + PyTorch + vLLM- 12mKeynote: How LLMs Have Changed the Way Organizations Do AI
21mLeveraging Ray to Enhance Machine Learning Models in Finance