Testing in Production – Mirroring CQL Traffic to a Shadow Cassandra Deployment
summary
A lightweight Rust TCP proxy duplicates CQL byte streams to per-node shadow clusters in Kubernetes, with online reconfiguration, per-session and global soft/hard limits, and Prometheus metrics. It handles thousands of sessions (about 150 MB per 10k sessions observed), supports multiple shadows, and adds negligible latency (typically <0.05 ms added latency). Buffering prevents primary stalls but drops slow shadows when limits hit. The system finds real bugs in pre‑release Cassandra (two discovered), runs in production since March, and will be open source.
AI-written summary · about
chapters (10)
- 0:00Testing in production intro
- 1:52Test real workloads
- 3:44Shadow deployment approach
- 8:26Rust TCP proxy
- 13:30Run shadow proxy
- 21:41Deploy in Astra DB
- 25:20Shadow deployment lessons
- 27:54Open source shadow proxy
- 28:51Q&A Rust performance
- 31:52Q&A internode traffic
related talks
24mMirror on the Wall, How Many K8s Environments Are Enough After All?
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
42mDistributed Data Stores on Kubernetes
26mTaking Your Database Beyond the Border of a Single Kubernetes Cluster
35mKubernetes Persistent Data Challenges – AZ, Region and Multi-Clou...
28mKafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replication
30mCatching Bugs Before They Bite: How Zillow Leverages Shadow Testing for Safer GraphQL Rollouts
05mIgnite - How Fast Was My Kubernetes?