Data and ETL with Notebooks in Papermill
summary
Explains a concrete system: papermill parameterizes notebooks and runs them headlessly, injecting JSON params, monitoring kernels, and writing immutable outputs to S3 or other sinks. Shows templates, CI tests, scheduler integration (Airflow), and cell tagging for docs and audits. Claims operational success at scale — Netflix runs tens of thousands of ETL jobs per day on this pattern — and presents the tools, failure modes, and fixes that make it usable in production.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:55Jupyter Notebooks overview
- 8:15Notebook limitations and Papermill
- 9:19Parameterizing notebooks
- 12:28CLI and Python execution
- 14:18Execution model and kernels
- 16:24Custom I/O handlers
- 17:07Productionizing notebooks
- 21:47ETL pattern templating
- 24:18Notebook best practices
- 26:04Breaking notebooks into DAGs
- 28:08Testing and debugging notebooks
- 30:34Papermill ETL advantages
- 31:39Netflix’s Papermill scale
- 32:52Q&A
related talks
26mWhat we learned from Papermill to operationalize notebooks
36mThe Lifecycle of a Jupyter Environment
34mNotebooker: Production and Scheduling for your Jupyter Notebooks
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
28mCreating custom engines for flexible notebook execution
33mData: The Land DevOps Forgot
25mTesting Data Pipelines
39mWhat Can Go Wrong With Testing an ETL Pipeline?