Observability and Debuggability in Cassandra
summary
Datadog SREs collect JMX, GC logs and request traces to tie latency spikes to resource signals and to validate fixes: long GC pauses from G1 humongous objects correlate with timeouts and disappear after increasing heap region size; a topology change triggers synchronized driver metadata reads that create a 4x request spike to ~30k req/s because 60k client connections per node all run range-slice queries; mitigation is either disable metadata queries or increase refresh window to 5m and spread client refreshes, plus monitor CPU, GC pause histograms and client-side metrics.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:25Cassandra outages debugging
- 2:50Cassandra cluster health basics
- 3:34Timeout errors CPU latency
- 6:23GC pauses causing timeouts
- 9:54Fixing humongous objects in G1 GC
- 11:18Graceful degradation CPU spikes
- 13:06Vertical scaling query surge
- 16:58Client connections range slices
- 19:05Range slice query solution
- 20:29Observability and metrics
- 21:32Debugging Cassandra issues
- 22:59Q&A
related talks
27mObserving Failure in Reactive Systems
12mUsing Dynatrace and Kibana to Identify Bad Queries and Optimise Them
52mIs your JVM app flying blind? Unmask hidden issues with Observability superpowers!
19mOops my app performance sucks... what if observability could be a game changer?
35mObservability for Dask in Production
41mScaling Observability: From Chaos to Clarity with Open Source Tools
47mHow I Solved Production Issues with OpenTelemetry (And How You Can Too)
33mObservability-First Kafka: Engineering Visibility at Scale