MLOps with Flyte
summary
Flyte is a Kubernetes‑native orchestration platform that claims reproducible, auditable ML pipelines by combining Flyte Admin control plane with Flyte Propeller operator in the data plane. The demo shows Flytekit Python code, typed tasks, dynamic workflows, and cache+recover behavior that skips cached nodes and restarts from failed nodes. It adds intra‑task checkpointing to resume long training epochs, supports GPU/spot machines and Spark via plugins, and surfaces runs in a web UI.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:37Software Engineering Practices
- 2:50Technical Debt
- 5:59Develop Incrementally and Constantly Iterate @ Scale
- 8:14Memoize, Recover and Reproduce
- 9:40Collaboration & Organizational Scaling
- 11:09Extend Simply
- 11:52Building Blocks: Tasks
- 12:16Workflows
- 14:37Workflow Modalities
- 15:36#1: Execute and Interact
- 17:37Features for Platform Folks
- 18:08Architecture Overview
- 19:44#1: Intra-task Checkpointing
- 24:00MLOps Best Practices
related talks
38mFlyte: Cloud Native Machine Learning & Data Processing Platform
39mFlyte: Production-Grade Data and Machine Learning Orchestration
30mEmbracing Multi-Tenancy While Scaling MLOps
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
26mEmbracing Multi-Tenancy While Scaling MLOps
25mTraining and Ensuring Reliability of ML Models at Wolt: The Power of ArgoCD, Flyte, and Argo Workflows
06mMLOps, Kubeflow, and Tekton