Intro to GKE setup of airflow with helm and terraform
summary
Provisions GKE with Terraform, deploys Airflow 2.5.3 using Helm charts, and runs Airflow on private VPC with Cloud SQL Postgres and a git-sync sidecar for DAGs. The platform hosts 88 active DAGs generating 1,200+ tasks per 24h, including single DAGs that spawn ~300 tasks from YAML-driven dynamic DAGs. Node pools autoscale 0→N for cost efficiency, KubernetesPodOperator creates per-task pods, and full cluster rebuild from Terraform plus Helm restores the deployment in ~45 minutes.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:57GitLab core values
- 3:00Agenda overview
- 3:22What is GKE
- 5:04Introduction to Apache Airflow
- 6:50What is Terraform
- 8:57Helm charts for Kubernetes
- 11:04GKE for Airflow
- 12:27Helm benefits for Airflow
- 13:51Terraform GKE model
- 14:12GKE Helm Terraform Airflow
- 18:28Airflow in GitLab data platform
- 20:57Benefits for data team
- 22:02Workflow dynamism with Airflow
- 23:06Kubernetes Pod Operators
- 23:49Terraform node provisioning
- 24:34Disaster recovery
- 24:56Setup best practices
- 26:42Monitoring and logging
- 27:23Q&A
related talks
25mRunning Apache Airflow at Scale
35mKubernetes journey
26mGo with the Flow: Automating Your Workflows with Airflow
36mApache Airflow in Production - Bad vs Best Practices
21mScaling and automating R workflows with Kubernetes and Airflow
18mETL at Scale: Optimizing PySpark + Airflow Workflows
01h 26mTutorial: Introduction to Kubeflow Pipelines
36mCyscale's Journey on Kubernetes