Airbnb Service Discovery: Past, Present, Future (Challenges of Change)

on YouTube ↗43m

" summary

Airbnb runs a homegrown SmartStack (Nerve + Synapse + HAProxy + ZooKeeper) for service discovery and moves it into Kubernetes, replacing Nerve with Mini Announcer and adding Envoy + SDS shim + mTLS. HAProxy reloads blow up with configs that list hundreds to thousands of backends (1,000+), causing memory exhaustion and thundering‑herd health checks; ZooKeeper packet rates climb from ~250k to ~400k pps under upgrade. Built an SD configurator, Cubegen, dashboards and a verification fleet; Envoy reduces memory and stale‑host problems but the migration and special‑case services (databases) remain hard.

AI-written summary · about

" chapters (33)

1. 0:00Intro
2. 1:48Smart Stack
3. 6:02MiniAnnouncer
4. 8:32OnWay Container
5. 11:03Scaling
6. 12:34Configs
7. 13:31Memory exhaustion
8. 14:59Network saturation
9. 17:50Moving to Envoy
10. 18:53Adding Egress Traffic
11. 23:03The Problem
12. 24:20The Question
13. 24:30Backend
14. 24:42Zookeeper
15. 26:24Invisible Limits
16. 27:43Connecting the Dots
17. 28:03Upgrading the Host
18. 28:20Upgrade Time
19. 28:52Solutions
20. 31:17Envoy
21. 31:45Smart SEC
22. 32:07Migration
23. 32:40Transparency
24. 33:00Dashboards
25. 34:20New Features
26. 37:08Adhoc Configuration
27. 38:22Multiple Sources of Truth
28. 39:38Service Version Rollout
29. 40:00Service Version Dashboard
30. 41:15Scale
31. 41:55Migrating
32. 42:25Thinking back
33. 43:03Outro

" related talks

• 46mExperiences from using discovery services in a microservice landscapeMagnus Larsson · Spring I/O 2017
• 25mHow to DDOS yourself with Envoy (and other tales of migration horror)Ben Plotnick, John Billings · EnvoyCon 2018
• 26mService Mesh Failure Stories at ScaleKrzysztof Słonka · ServiceMeshCon Europe 2020
• 32mAirbnb's Great Migration: Service APIs at scaleJessica Tai · apidays Singapore 2019
• 59mJava microservices from Netflix OSS to KubernetesAleksandr Nozdrin Plotnitskiy · JPoint 2019
• 29mThe rise of Layer 7, microservices, and the proxy war...Flynn · DevOpsDays Boston 2018
• 50mJourneys To Cloud Native Architecture: Sun, Sea & EmergenciesNicki Watt · GOTO Chicago 2019
• 34mIstio, We Have a Problem! Understanding and Fixing Bugs with a Service-MeshDavid Gageot · KubeCon + CloudNativeCon Europe 2019