Airflow and Kubernetes at JW Player, a match made in heaven?
summary
JW Player runs ~100 TB of analytics per month and replaces Airflow task code with Docker containers that run on Kubernetes via a custom operator that calls gantry to create jobs and posts lifecycle events to the fan‑out bus. Jobs fetch config from S3, read secrets from a Helm style YAML, must be idempotent to avoid duplicates, and use RDS for Airflow state and Snowflake for warehousing. The system even streams pod logs into the Airflow UI so orchestration stays centralized and observable.
AI-written summary · about
chapters (8)
- 0:00JW Player intro & data ecosystem
- 4:40Airflow overview & concepts
- 8:11Airflow use cases
- 12:08Airflow operator challenges
- 13:31Orchestration vs execution
- 16:46Kubernetes deployment system
- 20:39Custom Kubernetes job operator
- 22:52Benefits & adoption
related talks
21mScaling and automating R workflows with Kubernetes and Airflow
24mAutomating ML Pipelines With Kubernetes And Airflow- 21mScaling and automating R workflows with Kubernetes and Airflow
31mIntro to GKE setup of airflow with helm and terraform
28mScheduling Jupyter Notebooks Using Airflow on Kubernetes
29mHas Everyone Forgotten Application Workflows in Kubernetes?
38mBuilding and Managing a Centralized Kubeflow Platform at Spotify
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes