Supporting Reproducibility in Jupyter through Dataflows
summary
Claims a reproducibility model that makes execution deterministic by assigning persistent cell IDs and running cells bottom-up via a new DF kernel that recursively executes only needed dependencies. It supports multiple rich outputs per cell and user-defined output names so later cells reference named results instead of fragile numeric or underscore shortcuts. The prototype demos executing from a cold notebook, flags downstream stale cells, and shows interactive dependency graphs.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:50What is reproducibility
- 2:03Types of notebooks
- 3:00Rapid exploration
- 3:21Flexible environment
- 3:45Clear explanation
- 4:26Rapid exploration to clean explanation
- 4:57Jake Vander Plaats
- 6:00Notebooks
- 6:34Linking Cells
- 6:53ipython history
- 10:38Cell IDs
- 11:06Hex IDs
- 11:43Running cells
- 12:21Dataflows
- 12:56DF Kernel
- 18:10Multiple Outputs
- 18:55Other Projects
- 19:59Engineering
related talks
20mBuild an e2e analytics application using DataHub
39mEnabling Data centric Teams
16mMake your data science workflows reproducible
35mOrganizing data engineering around Kubeflow
28mScheduling Jupyter Notebooks Using Airflow on Kubernetes
32mReproducible Data Pipelines Using Controlled Effects
31mEvaluating fire simulators using Docker, Dask and Jupyter notebooks!
24mOrchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow