Accelerating AI training fleets with sched_ext
summary
Deploys a BPF-based userland scheduler sched_ext (scx_layered) across Reality Labs trainers with tens of thousands of GPUs to protect latency‑critical kernel‑launching and sync threads, detect them with kprobes/VRO and NVML heuristics, and roll out per‑job‑attempt so failures auto‑recover. Results show up to 9% GPU utilization uplift on some model families and fleet cost reduction. Work surfaces hard corner cases: CPU‑pinned tasks, IRQ noise, NUMA memory migration, and kernel tracepoint gaps.
AI-written summary · about
chapters (4)
- 0:00Introduction and agenda
- 1:18Theory and motivation
- 11:07Deployment and practices
- 22:55Lessons and future pain points
related talks
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
47mImprove Load Balancing With Machine Learning Techniques Based on the Sched_ext Framework
25mMaking the Most Out of Your Hardware Accelerators in a Kubernetes Cluster
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
21mUnlocking How To Efficiently, Flexibly, Manage and Schedule Seven AI Chips in Kubernetes
41mBeyond Uptime: Revolutionizing Fintech Reliability