Lessons Learned from Our Main Database Migrations at Facebook
summary
Facebook builds MyRocks, a MySQL storage engine on RocksDB, and migrates the UDB from compressed InnoDB to MyRocks, cutting storage roughly 50% space reduction and halving server count. Migration uses logical dumps, a fast bulk loader that bypasses WAL/memtables, online physical copies for instant replicas, and audit-replay to verify masters. Production issues include commit stalls tied to buffered I/O and kernel versions, secondary-index deletion corruption, slow empty-table range scans, and huge core dumps from allocator virtual memory.
AI-written summary · about
chapters (28)
- 0:00Intro
- 1:51MySQL at Facebook
- 4:11User Database (UDB) at Facebook
- 7:59What is My Rocks
- 10:09MyRocks Initial Goal at Facebook
- 12:17MyRocks features
- 15:07MyRocks cons vs InnoDB
- 18:22MyRocks migration - Technical Challenges
- 20:32Creating first MyRocks Instance without downtime
- 21:37Faster Data Loading
- 23:59Creating second MyRocks instance without downtime
- 24:43Shadow traffic tests
- 26:01Promoting MyRocks as a master
- 27:00InnoDB vs My Rocks, from SRE point of view
- 29:48Case a Stalls on COMMIT
- 31:13Hitting stalls on memory allocation
- 32:22Mitigating stalls
- 33:04Transition to Direct I/O
- 34:35A few deleted rows re-appeared
- 36:16Easy way to check index consistencies
- 38:29Scanning outbound delete-markers
- 39:00Reproducing
- 41:26Writing core files crashed servers
- 41:51Investigating logs
- 43:19Large virtual memory
- 43:58Workarounds
- 44:47Lessons Learned
- 46:15Our current production status
related talks
39mLessons Learned from Our Main Database Migrations
50mInnoDB to MyRocks Migration in Main MySQL Database
51mMyRocks deep dive and production deployment at Facebook
12mMyRocks at Facebook and a Roadmap
52mRocksDB Performance and Reliability Practics
57mRocksDB
27mMyRocks: best practice at Alibaba - MySQL Storage Engine
01h 01mPanel: Facebook's Distributed Database Evolution