Workshop: Deploy and Monitor ML Pipelines with Open Source and Free Applications
summary
Builds a reproducible pipeline that pulls US hourly electricity demand from the EIA API, handles the API 5,000‑row limit with pagination, and backfills six years (~half a million points per series) locally before scheduling hourly refreshes and 24‑hour forecasts via GitHub Actions using a multistage Docker image. It runs experiments (seven models), tracks parameters and best model per series with MLflow, profiles data with YData Profiling, renders a Quarto monitoring dashboard and deploys it to Quarto on GitHub Pages; free‑tier limits (no GPUs, variable scheduling) apply.
AI-written summary · about
chapters (11)
- 0:00Workshop introduction
- 1:47Tools overview and agenda
- 5:29GitHub Actions and Docker
- 16:53Five steps to deploy a machine learning pipeline
- 22:39Electricity demand forecasting
- 27:31Pipeline settings and parameters
- 29:47Backfilling historical data
- 31:59MLflow model training
- 34:14Deploy pipeline with GitHub Actions
- 39:13Dashboard and monitoring
- 41:00Q&A
related talks
01h 11mDeploy & Monitor ML Pipelines with Python, Docker and GitHub Actions
43mDemocratizing ML Pipelines using StreamPipes
33mDeploying and Scaling Spark ML and Tensorflow AI Models
01h 21mBuilding a ML Model Observability pipeline
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
49mOrchestrating data and ML workflows with Apache Airflow
44mML in Production
01h 23mBuild a production ML system with only Python on free serverless services