Optimizing Resilience and Availability by Migrating from JupyterHub to the...
summary
Migrating from JupyterHub to the Kubeflow Notebook Controller replaces a single-replica Hub API and in-memory CHP routing with a Kubernetes-native operator that stores state in etcd, uses Istio for traffic, and creates StatefulSet + Service per notebook; result: four times as many users, ten times concurrent executions, simpler control loops, fewer components, and noticeably lower operational overhead for platform engineers during load tests.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:05Jupyter notebooks history
- 4:38JupyterHub challenges
- 7:05QFlow and Kubernetes
- 10:38Notebook provisioning comparison
- 12:24Architecture and scalability
- 16:16Lessons learned
- 18:20Conclusion and Q&A
related talks
37mJupyterHub from the Ground Up with Kubernetes
31mKubeflow - Data Science on Steroids
27mDeploying JupyterHub with Kubernetes
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
31mIntroduction to Using and Use Cases of KubeFlow
28mScheduling Jupyter Notebooks Using Airflow on Kubernetes
39mHow We Hosted 200+ Data Scientists in a K8S Cluster
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes