Run Your Data and Machine Learning Workflows on Kubernetes with Flyte
summary
Flyte runs data and ML pipelines on Kubernetes and spins up and tears down ephemeral Spark clusters to run jobs without manual infra work. The Python SDK compiles workflows, automatically builds Docker images from an image spec, and schedules tasks as Pods with CPU GPU requests and limits. It supports Spark Ray Dask and distributed PyTorch (example: 2 workers), offers per-task caching so it returns cached outputs instead of rerunning, and scales to thousands of workflows (customers process 2.5 PB).
AI-written summary · about
chapters (14)
- 0:00Flyte and Union.ai intro
- 1:28Data and ML pipelines
- 3:14Pipeline challenges
- 5:21ML infra concerns
- 8:10Kubernetes for ML
- 10:18K8s adoption challenges
- 12:04Flyte workflow orchestrator
- 13:30Flyte core concepts
- 15:14Flyte uniqueness
- 16:37Building ML pipelines
- 19:48Registering Flyte workflows
- 21:34Optimizing with Flyte
- 24:04Flyte in model lifecycle
- 24:47Getting started with Flyte
related talks
38mFlyte: Cloud Native Machine Learning & Data Processing Platform
39mFlyte: Production-Grade Data and Machine Learning Orchestration
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
25mMLOps with Flyte
30mEmbracing Multi-Tenancy While Scaling MLOps
41mEfficient Data Parallel Distributed Training with Flyte, Spark & Horovod
28mFuture of Data Platforms: making use-case driven...
41mHow Spotify Leverages Flyte To Coordinate Financial Analytics Company-Wide