Scaling Erlang cluster to 10,000 nodes
summary
WhatsApp moves from FreeBSD/R16 to Linux/R21, packs servers into Open Compute blades, and scales a single Erlang distribution to about 1500-node cluster while reaching 10,000s via meta-clustering (multiple independent clusters connected through front-end chat nodes). Uses a central session manager plus PG2 with custom patches for high churn (600k–1.5M logins/sec), an extended distribution protocol (“andest”), and hotspot fixes: lock-counting BEAM builds, microstate accounting, BPF tracing and gdb etop scripts; many patches land upstream and most performance issues are resolved.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:55WhatsApp User Base Growth From 200m to 15B and growing fast
- 3:01Paradigm Shift Fow powerful servers to many tightly packed blades
- 3:57Foundation Replacement
- 5:36Erlang Cluster: Fully Connected Mesh
- 7:15Distributed Process Registry A good problem to solve
- 8:09Coordinated Approach (global)
- 8:40Distributed Hash Tables
- 9:26Keep It Simple More than one process registry
- 9:58Session Manager Central storage of phone-to-node mapping
- 10:57pg2 for Service Discovery
- 11:56Meta-clustering
- 14:36wandist: Extending Erlang Distribution Connecting disjoint Erlang clusters
- 16:32Challenges
- 19:10Diagnostic Tools
- 19:55Microstate Accounting
- 22:04Lock Counting & Source Code
- 25:18Erlang OTP is getting better Some original WhatsApp patches are no longer in use
- 29:12Embrace Open Source community
related talks
49mThat's 'Billion' with a 'B': Scaling to the Next Level
43mHow WhatsApp moved 1.5B users across data centers
53mIdioms for Building Distributed Fault-tolerant Applications with Elixir
40mReal Time Performance at Massive Scale
40mSo I used Erlang...is my system as scalable as they say it'd be?
39mAn adventure in distributed programming
30mHow to build distributed applications with Erlang
51mThe Erlang VM or How I Started Loving Parallel Progamming