Scheduling Jupyter Notebooks Using Airflow on Kubernetes
summary
Jupyter notebooks are run as on‑demand remote kernels on Kubernetes using an extended kernel spec (Docker image, CPU/memory, cluster/namespace, data access policies), notebook files are version‑linked, and scheduled via Airflow with a custom Airflow operator that extends papermill to connect a worker pod to a remote kernel. The system uses KubernetesExecutor, dynamic Spark allocation, namespace quotas and kernel culling to scale to hundreds or thousands of pods and deliver reproducible offline notebook pipelines.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 8:38Remote Kernel Execution
- 11:29Jupyter Notebooks
- 14:18Airflow
- 16:41Notebook Pipeline
- 19:15Pipeline Service
- 20:52Recap
- 21:34Upcoming features
- 22:59Questions
related talks
31mKubeflow - Data Science on Steroids
24mOrchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow
21mScaling and automating R workflows with Kubernetes and Airflow
06mLightning Talk: Using Jupyter Notebooks To Gain Insight Of Your Cluster
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
19mOptimizing Resilience and Availability by Migrating from JupyterHub to the...- 21mScaling and automating R workflows with Kubernetes and Airflow
31mIntroduction to Using and Use Cases of KubeFlow