Data Science at Scale: From Notebook to Production with Airflow and Ray
summary
Airflow now runs Ray jobs natively: add the ray_task decorator to a Python function and the task executes on a Ray cluster, returning a Ray object ID stored in XCom while it stores intermediate values in the Ray plasma store so large DataFrames stay in-memory between tasks. Alpha (May 2021) supports Modin and custom resources; beta adds checkpointing and an AnyScale connector; install via the provider and run distributed training, Tune searches, and Ray Serve endpoints from a single DAG.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 2:45Apache Airflow overview
- 8:21From notebooks to production
- 13:13Ray and Airflow integration
- 21:21Roadmap and next steps
- 22:47How to get started
related talks
29mScaling AI workloads with Ray & Airflow
31mNext-Generation Data Science Workflows Using Ray
35mOperating a Production-Scale Physical AI Data Platform with Ray
21mStreamlining AI Workflows with Apache Airflow and Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
24mBuilding Complex Data Analytics Pipelines with Ray
24mRay: A System for High-performance, Distributed ML Applications
30mHow to Scale Unstructured Data Processing with Ray Data