Deciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
summary
Bloomberg runs Cassandra as a managed service at 2.5 PB, 3,000 nodes, 250 clusters and shares concrete failures and fixes: mixed TTLs plus TimeWindowCompactionStrategy cause tombstone buildup; diagnosis uses sstable_expired_blockers and sstable metadata, then fixes include unchecked_tombstone_compaction or allow_unsafe_aggressive_sstable_expiration, or switching to leveled compaction (Unified compaction in Cassandra 5). Other incidents: native OOM leaves gossip alive (kill node or use JVM agent like jvm-quake), auth cache refresh timeouts (async refresh in 4.1), and imbalance fixed by matching replication factor to rack count and allocate_tokens.
AI-written summary · about
chapters (9)
- 0:00Cassandra at Bloomberg
- 2:05High disk usage and tombstones
- 8:47TTL and compaction fixes
- 9:49Cross-data center timeouts due to OOM errors
- 12:37Authorization errors under load
- 16:11Disk filling from imbalance
- 19:20Rack imbalance and replication
- 21:03Cassandra troubleshooting takeaways
- 22:05Closing remarks
related talks
23mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
53mApache Cassandra. Inception - all you need to know
29mCassandra - an introduction
45mCassandra Architecture: Understanding, Scaling and Optimizing
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
32mCassandra at Yammer #bbuzz
46mApache Cassandra: 8 years in production
35mReplacing Cassandra: A Digital Transformation for the World’s Largest Digital Identity Network