Deciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
summary
Bloomberg runs Cassandra at scale — about 3 PB across 3,500 nodes in 275 clusters — and finds concrete failure modes and fixes: mixed per-row TTLs with time-windowed compaction + mixed TTLs creates long-lived tombstones that pile up (remedies: unchecked tombstone compaction, allow_unsafe_aggressive_sstable_expiration when safe, or switch to leveled compaction); native native-thread OOMs leave nodes gossip-alive and break cross-DC handoff (kill the JVM, use JVM agents, fix tasksMax); short auth-cache validity (defaults 2s) causes spurious auth errors; multi-inserts create cell timestamps and bloated SSTables, fixed by LCS and tracing.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 1:56High disk usage and tombstones
- 10:02Timeouts and thread exhaustion
- 12:50Sporadic authorization errors
- 16:05Data size discrepancy after migration
- 21:24Key lessons and takeaways
related talks
25mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
37mPopular Apache Cassandra Problems & How to Fix 'Em
45mCassandra Architecture: Understanding, Scaling and Optimizing
29mCassandra - an introduction
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
35mThe What and Why of Cassandra Datamodelling at Scale
53mApache Cassandra. Inception - all you need to know
29mAnti-Patterns with Apache Cassandra