Troubleshooting your Data Workflows with Noteable+Dagster-live debugging of failed jobs
summary
A productionized Noteable notebook hooks into Dagster to become schedulable assets, with the workshop showing how to transition an exploratory notebook into a production job, materialize upstream Iris data as a reusable asset, and run notebooks via papermill so executed artifacts persist. The demo is mostly open source (95%) with a 5% UI piece, successfully executes notebooks, and proves the claim that one can dive into a live running notebook at the failure point with in‑memory state preserved to avoid costly re‑runs.
AI-written summary · about
chapters (20)
- 0:00Introduction and speaker welcome
- 1:08Why troubleshooting data pipelines is hard
- 3:15Overview of Noteable and Dagster integration
- 4:24Introducing the workshop and design process
- 5:27Task-focused vs asset-focused data pipelines
- 15:39How Dagster supports asset-focused pipelines
- 22:24Setting up the workshop environment
- 27:14Downloading and installing dependencies
- 34:06Tour of the workshop codebase
- 36:30Exploring the Jupyter notebook
- 38:12Creating the notebook asset in Dagster
- 41:16Launching Dagit and materializing the asset
- 47:42Refactoring the Iris dataset into a separate asset
- 52:16Updating the notebook to use the Iris asset
- 58:49Materializing the updated pipeline
- 1:00:55Uploading the notebook to Noteable
- 1:03:13Configuring the Noteable notebook asset in Dagster
- 1:11:04Materializing the Noteable notebook pipeline
- 1:16:07Live debugging of a failed notebook
- 1:19:44Q&A and closing remarks
related talks
01h 18mProductionizing Jupyter Notebooks
34mNotebooker: Production and Scheduling for your Jupyter Notebooks
39mNotebook To Production
28mUnlock the Full Potential of Jupyter Notebooks
01h 29mTurning Jupyter Notebooks into Data Applications
49mUntitled13.ipynb
25mFrom Data Exploration to Modeling to Deployment
27mThe Future of Notebooks in a Claude Code World