Online Database Reliability, Performance, and Consistency Engineering
summary
Operating at Meta scale uses single-primary, multi-replica clusters with sharded replica sets and a Tau cache in front of MySQL; UDB uses LSM/RocksDB to save space. Implemented consensus (Raft-like) fencing to avoid split‑brain; live promotions take under a second but full rebuilds of 1TB nodes take hours so fast recloning via GTID-based catchup is essential. Detection tooling includes replication checksums, block checks, full‑table comparison scanners that found a bit‑flip causing primary/replica drift and drove targeted replica resourcing.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:16Online Database Reliability, Performance and Consistency Engineering
- 4:26Sharding and Replica Set
- 6:45Multi Availability Zones
- 14:40Database Reliability Topics
- 18:16Avoid Split Brain
- 24:31Cascading Failover
- 31:12Common Database Overloads
- 34:00Identify bad queries
- 37:16Prevention
- 38:00Regressions on Database Upgrades
- 38:23Fleet Management
- 39:40Database Correctness with Replication
- 40:26Example of Correctness Expectations
- 41:27How databases can become inconsistent
- 42:02Can database find data errors automatically?
- 47:02A Primary-Replica data drift case
- 48:07Takeaways
related talks
52mData Consistency at Scale in Meta
29mDistributed SQL: How TiDB Modernized Data Management
47mLessons Learned from Our Main Database Migrations at Facebook
29mEnterprise Scale: How a Distributed Architecture Can Deliver Scale and Resilience
46mDatabase Reliability Engineering for MySQL
52mRocksDB Performance and Reliability Practics
39mLessons Learned from Our Main Database Migrations
21mDatabase Reliability Engineering