Why Airflow is a secret ingredient in MLOps
summary
Claims Airflow is the orchestration backbone for MLOps, with Astronomer contributing ~50% of commits and Airflow seeing ~9 million downloads/month. Real systems use Airflow + Snowflake, S3, Databricks, SageMaker, TensorFlow and KubernetesPodOperator to train weekly and score daily; one customer runs ~100 production models, another processes petabytes. Built features include OpenLineage integration (cross-platform lineage) and a notebooks-to-DAGs authoring preview; managed Astro service provides autoscaling clusters and multi-tenant workspaces that do work and recovery successfully.
AI-written summary · about
chapters (9)
- 0:00Introduction and personal story
- 3:33MLOps fundamentals and Airflow
- 7:07Airflow overview and Astronomer
- 10:19Data science team workflows
- 15:38Building Airflow DAGs
- 21:41Data science challenges
- 26:03Real-world ML workflows
- 31:53Why choose Airflow
- 32:18Q&A session
related talks
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
26mGo with the Flow: Automating Your Workflows with Airflow
43mHelp! My data is late!
35mBecome a Conductor with Apache Airflow
38mAirflow DAGs Made Easy: Introducing the Astro SDK
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
26mModern Data Orchestrators
30mPanel Discussion