8ms/99th write percentile latency - is it fast?
summary
Claims production Cassandra clusters hit 8ms 99th write latency at scale while serving ~300k req/s across ~1k EC2 nodes. Built self‑healing runbooks in RunDeck, moved monitoring from homegrown Hyperion/Sentinel to Datadog, and automated node add/remove, compaction and backups; SLOs use 99/95/65 percentiles and an error budget workflow. Result: developers stop getting woken at night and latency targets meet most SLOs, though upgrades and migrations remain slow and risky.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 3:23NoOps company and databases
- 10:17Reliability engineering principles
- 18:59DevOps misconceptions
- 21:12Cassandra basics and data modeling
- 26:36Setting up reliability at Codewise
- 34:25Monitoring and automation challenges
- 37:56Self-healing and operational procedures
- 42:38Outcomes and Cassandra improvements
- 44:30Q&A session
related talks
45m8ms/99th write percentile latency - is it fast? (...)
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
35mReplacing Cassandra: A Digital Transformation for the World’s Largest Digital Identity Network
39mMaking Cassandra Faster in Cloud-native architecture
57mDSS Keynote — Back to the Future: Prepare Your Data Infrastructure for Anything
27mReducing Client Latency and Timeouts When Running Cassandra on Public Cloud
46mCassandra Serving Netflix @ Scale
41mPerformance & Fault Tolerance: Building a Modern Database