Metadata Systems for End-to-End Data & Machine Learning
summary
Claims that metadata must describe entire ML systems not just model artifacts and shows concrete work: adoption of the open inference protocol and an open inference schema that standardize data-in-flight and metadata endpoints. Mentions integration with DVC and MLflow, servers like torchserve Kserve seldon core and Nvidia Triton, and contrasts complex metadata stacks like DataHub (elasticsearch neo4j MySQL Kafka). Demonstrates deployed Kubernetes pipelines producing interoperable metadata for discovery lineage and reproducibility and reports cross‑project adoption as evidence it works.
AI-written summary · about
chapters (8)
- 0:00Metadata systems for ML
- 2:39ML tooling evolution
- 8:03Metadata at scale challenges
- 13:39Model metadata limitations
- 17:10Deploying models in production
- 22:49Standardizing inference data with OIP
- 26:58Open inference schema
- 28:24Call to action for metadata standards
related talks
37mMetaOps: Metadata Operations For End-To-End Data & Machine Learning Platforms
28mThe Structure and Interpretation of ML Metadata
33mDiscover How to Create Your Own Metadata-Driven ML Platform from Scratch
46mJourney of metadata
24mMetadata driven framework for data discovery and data quality
28mthe semantic layer your AI needs
33mCompliant Data Management and Machine Learning on Kubernetes
37mMetadata Automation in the Data Mesh