Measuring and Optimizing Kubeflow Clusters at Lyft
Talkat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 16:25· Room 6C - San Diego Convention Center, Upper Level·
summary
Lyft builds Infraspend to scrape Prometheus from many Kubeflow clusters, persist metrics into Hive/Druid/Superset, and join those snapshots to AWS cost reports to attribute spending by namespace and pod. A surprising legacy footprint shows 518 xlarge EC2s in core clusters; using a CPU-allocation model plus autoscaler and pod-request tuning uncovers large unallocated capacity and reduces spend. The pipeline runs interpolation for missing hours and supports custom attribution schedules; the system is operational and informs budgeting.
AI-written summary · about
chapters (24)
- 0:00Intro
- 3:17What will you learn?
- 4:01What is the problem?
- 4:21Before Infraspend...
- 4:36Shipping Infraspend 2.0
- 7:28Kubernetes @ Lyft
- 8:36Notebook Instances
- 11:06Kubeflow Pipelines
- 12:10Hyperparameter Optimization
- 12:57How Does HP Tuning Work?
- 18:29Multi-tenant Platform Concepts
- 19:19Attribution Schedule Properties
- 19:50Attribution schedule generation
- 20:57Multi-tenant Platform Architecture
- 22:17Cost Models
- 23:08Infraspend is about allocation, not efficiency of that allocation.
- 23:54Mind the Unallocated Capacity
- 24:40Collecting Kubernetes Metrics
- 25:41Computing Allocation Schedule
- 27:11Challenges
- 28:50Migrating to Kubernetes
- 29:46Tracking Migration Impact
- 30:33Unallocated Cluster Capacity
- 33:16Mostly Open Source
related talks
29mKubeflow Trainer Observability: Real-Time Progress Tracking for Reliab...
40mOptimizing AI/ML Workloads
53mImplementing Kubernetes on with confidence
17mKepler: Powering up the efficiency of your cluster
50mFast-track Kubernetes and platform rebuilds with observability
25mKube Intelligence - A Metric Based Insightful Remediation Recommender
34mAutoscale your Kubernetes Workload with Prometheus - Frederic Branczyk, CoreOS
42mMetrics at scale