AI pipelines powered by Jupyter notebooks
summary
Shows how multiple Jupyter notebooks become reproducible pipelines by composing them with concrete tools and demos: use of run magic, nbconvert to produce HTML, papermill for parameterized execution and programmatic APIs, an Airflow operator to embed notebooks into DAGs, and Kubeflow pipelines to compile, run experiments on Kubernetes with container images and GPU allocation. Demo generates CSVs from GitLab repos, converts an overview to HTML, stores artifacts in object storage, and returns an executed notebook and HTML output.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 4:21Jupyter notebooks overview
- 7:53Building analytics pipelines
- 12:14Simple pipeline tools
- 14:21Advanced execution with Papermill
- 16:48Demo: notebook pipelines
- 22:19Enterprise analytics challenges
- 23:46Apache Airflow for pipelines
- 26:37Deep learning isolation
- 31:41AI platforms on Kubernetes
- 34:51Kubeflow Pipelines
- 36:08Demo: Kubeflow with notebooks
- 39:01Q&A
related talks
42mSimplifying Data Analysis with GitHub Codespaces, Jupyter Notebooks & Open AI
54mThe secret weapon for developers, Notebooks.
39mJupyter AI — Bringing Generative AI to Jupyter
32mCombining IPython with Open Source Papermill to enhance your Jupyter Notebook
01h 04mAI Agents and Jupyter Notebooks for Security Data Analysis
34mOrchestrating Generative AI Workflows to Deliver Business Value
27mAI-Powered ETL Pipeline Orchestration: Multi-Agent Systems in the Era of Generative AI
26mWelcome to the AI Jungle! Now What?