Future of Data Platforms: making use-case driven...
summary
Lyft runs an open source platform (Flyte) that treats ML as a superset of data, offering serverless orchestration across Kubernetes clusters, Java/Python SDKs, and automatic wiring to Spark, SageMaker and TF operators. It records immutable, versioned workflows and a versioned artifact store, supports memoization to skip expensive recomputations, and claims reproducible replays. At Lyft it handles over 1 million pipelines per month and >40 million containers on Kubernetes, and it enables transparent vendor swaps without user disruption.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:00Agenda
- 1:45Data Platforms
- 2:31Lift example
- 3:53What is a data platform
- 6:49Flight
- 7:51User requirements
- 11:13User workflows
- 12:50Experimentation
- 15:15Iterating
- 17:13Production
- 17:52Modern
- 18:52Extensibility Flexibility
- 20:22Assumptions
- 22:38What is Fuze
- 23:28Fuze User Journey
- 25:13Fuze Concepts
- 25:28Version
- 25:54dynamism
- 26:21scale
- 26:45design
- 27:46ecosystem
- 28:04outro
related talks
38mFlyte: Cloud Native Machine Learning & Data Processing Platform
04mInterview
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
39mFlyte: Production-Grade Data and Machine Learning Orchestration
28mEnabling Machine Learning with Apache Flink
09mPlatform Engineering for Data Science at Scale
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink