Discover How to Create Your Own Metadata-Driven ML Platform from Scratch
summary
Metadata-driven ML platform implemented on Kubernetes combines Presto + Apache Iceberg for an open lakehouse and Kubeflow Pipelines for orchestration, plus Katib for HPO, training operators for distributed training, and KServe for serving. Demo shows federated SQL joins across MySQL and MongoDB, Iceberg ACID/time-travel for reliable feature tables, a Presto SQL web client and Jupyter integration, and an automated KFP notebook pipeline that runs, caches steps, and exposes model predictions via KServe; demos are recorded due to network instability.
AI-written summary · about
chapters (8)
- 0:00Metadata-driven ML platform intro
- 1:28Open Lakehouse as foundation
- 8:16Key components
- 15:44Demo: Presto and Iceberg
- 21:06Automating ML with KubeFlow
- 25:27Demo: KubeFlow pipeline
- 28:29Summary and open-source
- 29:20Q&A
related talks
33mCompliant Data Management and Machine Learning on Kubernetes
29mMetadata Systems for End-to-End Data & Machine Learning
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
46mJourney of metadata
36mEnterprise-grade Full Stack ML Platform
22mBuilding a MLOps Platform with Kubeflow on GKE
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes