Reproducible Scalable Workflows with Nix Papermill and Renku
summary
Combines Nix for deterministic, hashed environments with Papermill to run parameterized notebooks and Renku to version data and emit CWL; demonstrates reproducible runs on laptops and HPC where Docker is unavailable, warns that Nix builds are slow initially and Jupyter’s Node+Python stack resists fully pure packaging, and shows how to script environments so users get identical Python setups via Nix shells and exported Conda configs. Nix store hashes Papermill notebooks as functions Renku generating CWL
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:12Outline
- 1:45Standard Approach
- 2:48Modern Data Analysis
- 5:18Reconciliation
- 6:19Pure Python Packages
- 6:29Distributions
- 7:06Pip Requirements
- 8:38System Dependencies
- 11:37Installation (Multi-User)
- 12:38An Aside into Purity
- 13:43Nix Python Expressions
- 15:07Replacing Conda II
- 16:17What Reproducibility?
- 16:46Data Science Woes
- 17:42Cluster Woes
- 20:49Xeus Python
- 21:16Reusing Notebooks
- 24:29Parting Practicalities
- 26:20Conclusions
related talks
30mReproducible work environments for data scientists using Nix
59mReproducable environments using nix
05mReproducible Build Environments with Nix
29mReproducible Python projects using Nix
58mFunctional Infrastructure With Nix - Part 1 of 2
31mNix: Need an Ultra-Robust Dev Environment? Create an Immutable, Reproducible Env
24mA sack full of angry snakes: Taming your python dependencies with Nix
36mFunctional programming and Nix for reproducible, immutable infrastructure