End-to-End ML Pipelines with KubeFlow and TensorFlow Extended (TFX)
summary
Live demo of end-to-end pipelines that actually run on a Kubernetes cluster using Docker images, Airflow or Kubeflow as orchestrators, TFX components for TFX data validation and feature transform, Apache Beam for schema and statistics, MLflow for experiment tracking, Papermill to parameterize notebooks, Katib for hyper-parameter tuning and TF Serving + Istio for canary traffic. Shows pipeline runs and logs on the cluster, compares metric snapshots across runs, and applies TensorFlow Privacy (differential privacy) to trade accuracy for a measurable privacy budget.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 10:28ML pipeline systems overview
- 23:41TFX components and validation
- 25:08Kubeflow live demo
- 31:48MLflow tracking demo
- 33:18Papermill and notebook execution
- 34:26Model privacy with TensorFlow Privacy
- 36:37Q&A and resources
related talks
29mML Engineering Kubeflow
38mTensorFlow Extended (TFX): Machine Learning Pipelines and Model Understanding
39mDeploy your ML Workloads on Kubernetes with Kubeflow
22mThe Containerization of Machine Learning: TensorFlow: Kubernetes, and Kubeflow
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
34mBuilding End-to-End Machine Learning Workflows with Kubernetes...
31mKubernetes for Machine Learning: Managing and Scaling AI Workloads
14mKubeflow: Where ML Meets Kubernetes