Scaling and automating R workflows with Kubernetes and Airflow
summary
UK Health Security Agency builds a platform that runs R and Python workloads on HPC using Kubernetes, OpenShift, Docker, git and Airflow, claiming it lets analysts stop worrying about laptop memory and process millions of COVID records. Each Airflow DAG launches a pod per task with container images containing code and credentials, teams map to team namespaces, CI/CD pushes images from GitLab/GitHub, and operational reporting shows the setup works in production, saving staff time and reducing failed ad hoc runs.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:23Farm analogy
- 1:40Who am I
- 3:30Agenda
- 4:02Context
- 7:31Where we came from
- 8:34What is Kubernetes
- 12:53What is Airflow
- 15:32How have we used these together
- 18:30How to get started
related talks
21mScaling and automating R workflows with Kubernetes and Airflow
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
24mAutomating ML Pipelines With Kubernetes And Airflow
35mBuilding a Medical AI with Kubernetes and Kubeflow
31mIntro to GKE setup of airflow with helm and terraform
25mRunning Apache Airflow at Scale
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
54mScaling Machine Learning Pipelines in Cloud