Building Self-Serve, Multimodal Medical Image Pipelines with Ray
summary
Heartflow builds PAL: a self-serve pipeline that converts S3 blob scans into Iceberg tables via a bronze/silver/gold medallion, using Ray Data for distributed IO and compute, Dagster assets for orchestration, and Anyscale autoscaling on spot instances. It handles petabyte backfills and nightly jobs, vectorizes row transforms with PyArrow, and pivots from pyiceberg commits to publish to Iceberg via Athena after driver OOMs. CI/load tests, peta‑scale backfills, and an AI debugger (RoboPAL) complete the production stack.
AI-written summary · about
chapters (17)
- 0:00Introduction and HeartFlow overview
- 2:53PAL pipeline goals and requirements
- 4:20Ray as the core system choice
- 5:44High-level architecture overview
- 8:12Bronze layer execution with Ray Data
- 9:57Silver and gold layer processing
- 11:45Publishing to Iceberg and Athena
- 12:47Orchestration with Dagster
- 14:34Dagster-Ray integration via S3
- 16:24Developer experience and PAL SDK
- 18:12Observability with RoboPal
- 19:37Scaling Ray for large workloads
- 21:10Debugging Ray jobs with RoboPal
- 24:19Ray’s role in unlocking data discovery
- 25:48Comparison with Spark and operational benefits
- 26:11Data sources and medical imaging context
- 28:51Q&A session
related talks
24mBuilding Complex Data Analytics Pipelines with Ray
29mScaling ML Pipelines for Autonomous Driving with Ray
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
28mProcessing medical images at scale on the cloud
01h 04mBuilding Efficient and Frugal Image Data Processing Pipelines
25mData Science at Scale: From Notebook to Production with Airflow and Ray
31mHow Rad AI is Revolutionizing Radiology with Ray
46mScaling ML Pipelines with Feast, Ray and Kubeflow