Generative AI Model Data Pre-Training on Kubernetes: A Use Case S...
summary
Demonstrates a production pipeline that uses Kubeflow Pipelines, KubeRay and Ray to run embarrassingly parallel annotative transforms (deduplication, language ID, PII and toxicity checks, tokenization) with per-worker S3 IO, checkpoint/recovery and task-specific Ray clusters. An example fuzzy-dedup job processes 8.5 billion documents (23 TB compressed) down to 5.6 billion (14 TB) on a 7,500‑core, 56 TB RAM cluster over ~40 hours. An open source Data Prep Kit (DPK) standardizes components and runs nested KFPv2 pipelines; UI and reproducible runs are used for operationalization.
AI-written summary · about
chapters (12)
- 0:00Generative AI pre-training intro
- 1:03Data processing workflows
- 3:11Scaling with Ray on K8s
- 5:59Ray cluster management
- 7:00Kubeflow Pipelines orchestration
- 10:16Demo: Kubeflow + Ray
- 13:04KFP v1 vs v2 comparison
- 16:58Production Ray data processing
- 17:20Fixing data processing inconsistency
- 18:46Data Prep Kit introduction
- 19:52Summary and resources
- 20:34Q&A on workflow engines
related talks
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...
34mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
33mCompliant Data Management and Machine Learning on Kubernetes
35mBuilding a Medical AI with Kubernetes and Kubeflow
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes