Scaling Python to thousands of nodes with Ray
summary
Claims ability to run Python workloads across thousands of nodes using Ray, demonstrating actual code changes and autoscaling from a laptop to a cluster that briefly hits ~24 CPUs + 1 GPU and back. Uses Ray Data for streaming ingestion, Ray Serve for model endpoints, embeddings + vector DB + a reranker and a larger-context M7B-style LLM. Builds a RAG for the Wheel of Time, ingests EPUBs in ~20 minutes, reduces hallucinations but still needs tuning to be spoiler-safe.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:56Wheel of Time character tool
- 3:39About Ray
- 5:44Distributed computing challenges
- 7:27What Ray does
- 9:11Ray cluster architecture
- 10:14Ray tasks and actors
- 13:23LLM query to RAG pipeline
- 15:29Ray Data for streaming
- 17:38Live demo: Ray Data scaling
- 19:05Ray Serve for model serving
- 21:33RAG accuracy with chunking
- 24:25Live demo: RAG service
- 26:52Project takeaways
- 28:20Q&A
related talks
26mRay: A System for High-performance, Distributed Python Applications
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
57mIntroduction to Ray for distributed and machine learning applications in Python
02h 51mDistributed Python with Ray Hands on with the Ray Core APIs
29mLearning to Scale Data Science, Machine Learning, and Pandas with Ray and Modin
39mScaling Python for Machine Learning: Beyond Data Parallelism
40mScaling Python - Bank Edition