Building a Data Infrastructure for AI/ML
summary
Claims that Open Table Formats Apache Iceberg Apache Hudi Delta Lake make object storage a full data warehouse and show a reference architecture that supports MLOps, distributed training, and generative AI. Implements MinIO as storage, uses MinIO + MLflow + Ray + Kubeflow/MLRun for pipelines, checkpoints every epoch, and proves distributed training on a dataset too large for memory works and speeds training. Notes zero‑copy branching for feature engineering, recommends 100→200Gb networks for modern GPUs and mentions Blackwell‑class petaflops and an enterprise MinIO cache.
AI-written summary · about
chapters (13)
- 0:00Data infrastructure for AI/ML
- 1:26Data lakehouse benefits
- 7:29AI/ML workloads in lakehouse
- 7:50MLOps importance
- 9:17MLOps tools comparison
- 13:53Traditional vs generative AI
- 15:59Document pipeline for generative AI
- 17:44Vector databases and search
- 19:28Distributed training MinIO Ray
- 21:15GPU performance trends
- 25:52MinIO cache for GPUs
- 27:39GPU metrics fun facts
- 28:21Q&A
related talks
24mObject Storage Driven Machine Language Workloads
35mWriting Machine Learning Pipelines Against Object Storage
35mHigh-performance data foundation for AI success
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)
33mBuilding Resilient (ML) Pipelines for MLOps
41mInfrastructure for ML Applications
11mBreaking Data Silos: Enterprise Data Management
44mWhat's the Best Big Data Architecture for You?