From Chaos to Clarity: Deciphering Cache Inconsistencies in a Distributed...
summary
A distributed EVCache deployment spans 194 clusters and 22,000 servers storing 2 trillion items (14.3 PB) and its replication engine handles 30 million replication ops/sec. A global launch stalls because delete events land as double-hashed keys in the target region and return “delete not found.” The root cause is a library change that enables key hashing by default; short-term mitigation is setting replication.hash_key=false, long-term work is aligning library defaults and adding end‑to‑end tests and production shadowing.
AI-written summary · about
chapters (16)
- 0:00Introduction and context setup
- 1:04The unexpected holiday gift of January 2023
- 2:06What is EVCache and its architecture
- 3:32Fleet overview and replication operations
- 4:15Replication service modes and mechanisms
- 5:59The critical discrepancy on January 13th
- 6:41Impact on password sharing feature launch
- 7:23Debugging journey begins with usual suspects
- 8:28Key detail from app team about library upgrade
- 9:11Breakthrough with shadow environment and key hashing
- 10:36Root cause: double hashing in new library
- 11:37Fix implementation and results
- 11:57Mitigation strategies and long-term fixes
- 12:4148-hour ordeal and team collaboration
- 13:44Lessons learned and future improvements
- 14:48Closing remarks and Q&A
related talks
38mSecuring Distributed Cache: Achieving Secure-by-Default with Key Challenges &...
43mCaching half a billion user transactions
53mLeveraging Caching For Cloud Native Applications
46mEdge Performance with In memory NoSQL
25mMigrating Datastores
21mDistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed Caching
15mWhy We Have 17 Clusters Instead of One (and Why You Probably Shouldn’t Do Th...
54mDistributed Caching in Real-Time Payment Systems