Data Products, From 0 to data science pipeline

on YouTube ↗34m

" summary

Starts with a working prototype in Jupyter notebook then parameterizes it with papermill to produce many variants, uses Apache Airflow DAGs to grid‑search hyperparameters (example loop 2 to 20 step 2), short‑circuits when a 97% threshold is met, runs canary comparisons and deploys the winner. Orchestrator scales via Kubernetes executor or Celery, launches Docker images per task, claims support for thousands of parallel notebooks and a custom RapidFlow operator that submits training pods to Kubernetes.

AI-written summary · about

" chapters (9)

1. 0:00Minimum viable data products
2. 1:54Defining minimum viable data products
3. 3:39Jupyter notebooks for data science
4. 7:58Scaling Jupyter with Hub/Binder
5. 10:59Productionizing models with Airflow
6. 15:48Hyperparameter tuning in Airflow
7. 21:06Scaling Airflow with Kubernetes/Celery
8. 27:29Deploying Airflow locally/production
9. 30:09Summary and next steps

" related talks

• 16mBridging Data Science and Data Infrastructure with Apache AirflowDaniel Imberman · PyCon US 2020
• 17mProductionizing Data Science WorkloadsAnand Iragavarapu · PyCon India 2020
• 49mTutorial: From Notebook to Kubeflow Pipelines to KFServing: the Data Science OdysseyKarl Weinmeister · KubeCon + CloudNativeCon North America 2020 Virtual
• 32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & KubernetesSavin Goyal, Saravanan · KubeCon + CloudNativeCon North America 2022
• 25mFrom Data Exploration to Modeling to DeploymentRoope Astala · O’Reilly Jupyter Pop-up Boston 2018
• 49mOrchestrating data and ML workflows with Apache AirflowTamara Janina Fingerlin · Conf42 Machine Learning 2023
• 39mOperating Deep Learning Pipelines Anywhere Using KubeflowJörg Schad, Gilbert Song · KubeCon + CloudNativeCon China 2018
• 20mBuild an e2e analytics application using DataHubTom Coufal, Michael Clifford · DevConf.CZ 2021