Living with the Pathology of the Cloud: How AWS Runs Lots of Clusters

on YouTube ↗35m

" summary

EKS control planes run on public AWS primitives: API Gateway + Lambda front end, CloudFormation stacks per cluster, a dedicated VPC, NLBs, and managed node groups; etcd runs as three-node clusters per control plane with managed etcd backups and quorum monitoring. Operators hit a 15 minutes TCP stall caused by a Go HTTP/2 connection bug that left connections established, and a mutating webhook pushed p99 pod creation for 5,000 pods from ~5m to 46m; adding an in-memory cache restores sub-second P99s. Cellular architecture isolates rollouts into many shards to limit blast radius.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 1:14EKS Service Priorities
3. 5:25Managed Node Groups
4. 6:05Cellular Base Architecture
5. 8:24Tradeoffs
6. 10:02How is the cluster created
7. 13:48Kubernetes learnings
8. 20:18CD quorum
9. 26:51Service accounts
10. 29:01Data Path
11. 32:26Lessons Learned
12. 33:32Questions

" related talks

• 35mKubernetes on AWS: Practices & OpinionsArun Gupta, Raffaele di Fazio · KubeCon + CloudNativeCon North America 2017
• 17mKeynote: Kubernetes: Lessons from the TrenchesArun Gupta · KubeCon + CloudNativeCon + Open Source Summit China 2019
• 17mKeynote: Kubernetes: Lessons from the TrenchesArun Gupta · KubeCon + CloudNativeCon China 2019
• 11mAmazon EKS RoadmapEswar Bala · AWS Container Day 2019 San Diego
• 27mRunning Kubernetes in Production A Million Ways to Crash Your ClusterHenning Jacobs · Container Camp UK 2018
• 16mKeynote: Cloud Native at AWSAdrian Cockcroft · KubeCon + CloudNativeCon North America 2017
• 37mkubernetes on AWS gone wildChristian Jantz · code.talks 2017
• 29mAmazon Elastic Container Service for Kubernetes (EKS): A case study - M. Re Ferre' and G. CarabelliM. Re Ferre', G. Carabelli · ContainerDay 2018