How WhatsApp moved 1.5B users across data centers
summary
WhatsApp moves 1.5B users by replicating state to Facebook data centers, building a custom TCP protocol and a C++ gateway to talk to Facebook services, implementing Thrift on Erlang, and using a gen_factory with sticky queue scheduling. A pipeline of DB replay, DB switch and DB crawler makes repairs idempotent, rate‑limited, backoff‑aware and peer‑replicated. Country‑level read‑only windows let traffic cutover finish in under five minutes; no persistent data‑errors appear during the migration.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:07Server Architecture
- 3:09Reasons to Migrate
- 4:50Migration Plan
- 10:40Migration Preparation
- 16:14What If an error happen? db replay
- 18:20What if too many errors happen? db_replay feedback
- 20:36What if a key is hot?
- 21:31What if repairs return error? db_replay backoff
- 23:27What if a node loses disk? db_replay replication
- 24:44What if db_queuer discarded? db dirty keys
- 26:08What if a node crashes? db dirty lag
- 27:26How to validate data? compare
- 28:32How to deal with flaky compare? db compare
- 29:39How to copy offline data?
- 30:14How to validate offline data?
- 30:58How to move users to the other side? db failover
- 33:34What's there on the other side? same thing!
related talks
16mRelay Infrastructure at WhatsApp Scale
49mThat's 'Billion' with a 'B': Scaling to the Next Level
35mScaling Erlang cluster to 10,000 nodes
24mMySQL for Messaging
14mState of the Union
43mMulti-tenant HBase Solutions at Facebook
25mMass Messaging: Reaching Millions During Large-Scale Disasters
22mCross Atlantic: Scaling Instagram Infrastructure from US to Europe