The Lifecycle of a Jupyter Environment
summary
A single notebook grows into a runnable ETL pipeline with an actual ETL.py class that registers steps for extract validate clean and execute, backed by tests and CI and health checks. It uses Papermill parameterized runs to automate repeated notebooks, MLflow experiment tracking to version models, and Snowflake or SageMaker for deployment with Rapids for GPU acceleration. The talk cites handling schema drift and evolving labels, and a 23,000‑respondent survey as context for tooling choices.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:08Speaker and community
- 4:01Objectives for notebooks
- 8:34Modular notebooks architecture
- 13:38Choosing Jupyter tools
- 17:51Reproducible packaging challenges
- 22:08Deploying resilient projects
- 26:49Q&A
related talks
39mData and ETL with Notebooks in Papermill
37mPatterns for Collaboration Between Data Scientists and Software Engineers
42mBuilding ML Pipelines in JupyterLab Using Elyra - Without the Need to Write Code
01h 23mImproving Your Data Modeling Work Through Open-Source Software
31mPanel: Jupyter in Education
01h 28mMaintainable code in data science
24mJupyter Notebooks for Ops
25mTo Notebook or Not to Notebook: Multilingual Workflows for Data Analysis