From Agents to Action: Scaling Agentic AI with Ray on Intel Xeon
summary
Describes deploying agentic pipelines that push tool execution and retrieval onto Intel Xeon 6 CPUs while using Ray for distributed orchestration and parallel tool invocation, claiming Intel Xeon 6 hosts small LLMs for cheaper inference, a router that routes between CPU-hosted SLMs and cloud frontier models, measured 1.6–1.7× tool speedup when offloading GIL-bound calls to Ray workers, an example 64s CPU / 62s GPU split for a 70B model, reported 27.2% cost savings and 85–95% cosine similarity with frontier outputs and a 1.44× user capacity increase.
AI-written summary · about
chapters (11)
- 0:00Introduction to agentic AI
- 1:32Market shift to inference and agents
- 3:22Token economics in agentic workloads
- 5:51CPU role in agentic pipelines
- 7:43Open-source integration with Ray
- 8:46End-to-end agentic pipeline design
- 11:53Offloading tools to Ray workers
- 12:37Live demo of smart router
- 13:41CPU-GPU ratio for token economics
- 15:47Optimizing user capacity with CPUs
- 16:29Closing remarks and resources
related talks
29mBuilding Scalable AI Agents with Ray
15mRay: A Distributed Compute Engine for AI
04m4 real-world AI deployments at enterprise scale
28mScaling Multimodal Data and Reinforcement Learning with Ray
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
15mRay: A System for Distributed Applications
31mFaster Model Serving with Ray and Anyscale
28mOptimizing GenAI with Intel Gaudi Accelerators on