Scaling AI workloads with Ray & Airflow
summary
astro-provider-ray (Apache-2) claims to let Airflow create and delete Ray clusters on Kubernetes, submit Ray jobs and monitor progress in the Airflow UI, accept dynamic cluster configs, export results to XCom, and offer a TaskFlow decorator that hides setup/teardown. Live demo shows cluster creation and successful job submission for simple examples (hello / square) while cluster deletion hits a known bug. Adoption is early: ~1k downloads/day (~<20k/month) versus Cosmos ~16M/month and Airflow ~20–25M/month.
AI-written summary · about
chapters (15)
- 0:00Gratitude to conference organizers and personal note
- 1:06Airflow and BBC recommendations
- 4:14Why Airflow for BBC
- 6:42Open-source at Astronomer
- 8:08Airflow 2 features
- 10:38Ray for data processing
- 11:41Ray architecture
- 14:12Ray with Airflow
- 18:45Demo setup
- 20:12Ray cluster in Airflow
- 23:29Cleanup and syntax
- 24:51Use cases and feedback
- 26:17Adoption and future
- 27:22Provider inclusion discussion
- 28:45Q&A
related talks
25mData Science at Scale: From Notebook to Production with Airflow and Ray
21mStreamlining AI Workflows with Apache Airflow and Ray
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
35mOperating a Production-Scale Physical AI Data Platform with Ray
31mNext-Generation Data Science Workflows Using Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
31mHosting AI models with Ray and GKE