Scaling Redis writes with cluster mode
summary
Zendesk migrates a write‑heavy workload to Redis Cluster, hits cross‑slot pipelining and transaction failures, and fixes them by upgrading to redis gem v5 and contributing code into redis-cluster-client gem (multiple small PRs plus a temporary fork deployed). The changes add cross‑slot pipelining, proper WATCH/EXEC behavior for transactions, and connection/retry fixes; production tests show stable scaling across added shards and a ~3–4 ms P99 write latency improvement.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:56Redis deployment options
- 7:05Sentinel and scaling limits
- 9:12Cluster Mode for writes
- 13:09Analyzing scaling incident
- 17:45Fixing cross-slot errors
- 20:13Transactions in cluster mode
- 23:43Contributing to Redis gem
- 28:38Open source lessons
- 32:10Key takeaways and next steps
related talks
59mUsing Redis beyond caching
01h 02mBreak Out of Redis
51mX-Raying Your Redis App: SDK Tuning with a Performance Engineer’s Eye
25mMigrating Datastores
50mScaling Postgres With Some Help from Redis
56mRedis Application Patterns in Rails
36mBuild a Cloud Native Asynchronous Messaging System For Scale With Redis
46mFun with Ruby and Redis