From OOMs to 2x Throughput: Scaling Multimodal Data Curation on Ray
summary
Shows a concrete pipeline that extracts camera, lidar, and vehicle state from terabytes of proprietary logs using Ray Data streaming plus Ray Core task isolation, isolates C++ failures by running extractors as separate processes with max_calls=1, recovers hung tasks via ray.wait with num_returns=1 and timeout and explicit sigkill, records failures to a centralized ray.util.queue and reports metrics to Prometheus, and delivers runs of >7M samples on 200 nodes at over 2x previous efficiency (example speedups: a 7M run drops from 5d19h to 2d4h, a 300k job from partial 14h to 5h).
AI-written summary · about
chapters (19)
- 0:00Introduction and agenda
- 1:10Workload and data challenges
- 2:55C++ bindings and memory issues
- 5:30Redesigning the workflow
- 7:15Isolating C++ code with Ray
- 9:44Avoiding head-of-line blocking
- 11:52Killing unresponsive tasks
- 12:54Streaming pipeline design
- 14:38Tracking failures with RayQ
- 16:48Failure analysis and reporting
- 17:51Custom metrics reporting
- 20:25Cluster resource tuning
- 21:52Block size and write delays
- 23:37Balancing block sizes
- 24:59Handling skewed workloads
- 26:45Schema normalization
- 27:26Performance results and stats
- 28:31Key takeaways and summary
- 29:27Q&A session
related talks
26mData Processing on Ray
30mHow to Scale Unstructured Data Processing with Ray Data
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
22mLarge Scale Data Loading and Data Preprocessing with Ray
24mBuilding Complex Data Analytics Pipelines with Ray
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
35mMars-on-Ray: Accelerating Large-scale Tensor and DataFrame Workloads
25mData Science at Scale: From Notebook to Production with Airflow and Ray