Hosting Notebooks for 100,000 Users
summary
Quanto peen integrates the Jupyter UI at web scale by replacing filesystem state with a drop‑in PGContents that stores notebooks as blobs in Postgres (using Postgres TOAST), authenticating users via JupyterHub + OAuth, and scaling across many Hubs with an iframe-based proxy and allocator. It adds NB/server extensions for sharing, uses PGbouncer to avoid connection exhaustion, and supports ~65,000 notebook authors, ~220,000 notebooks, ~31,000 checkpoints and ~450 GB of stored notebooks in production.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:19Outline
- 4:48Scale
- 8:03Jupiter Server
- 9:10Jupiter Hub
- 10:49Configurable HTTP Proxy
- 11:21User Identity
- 12:30OAuth
- 12:53Notebook Storage
- 13:14File System Manipulation
- 16:02PostCrisis Solution
- 19:13Vanity Metrics
- 19:38PostgreSQL Toast
- 20:43Multiple Hubs
- 21:05Python
- 21:24Indirection
- 22:37Embedding
- 23:29Implementation Notes
- 23:43Sharing notebooks
- 24:14Notebook extensions
- 24:25NB extension
- 25:19General notes
- 26:12Conclusions
- 26:54State is the enemy
- 27:11A throne of lies
related talks
19mJupyterHub: A "things explainer overview"
33mIntegrating Jupyter Notebooks into your Infrastructure
34mMoving notebooks into the cloud: challenges and lessons learned
32mSpin up Jupyter notebooks at scale and enhance your productivity
37mReproducible and shareable notebooks across a data science team
01h 36mDeploying Jupyter Notebooks for students and researchers
34mNotebooker: Production and Scheduling for your Jupyter Notebooks
24mJupyter Notebooks for Ops