Building an Efficient Replication System for Thousands of Terminals
summary
Describes a replication architecture that delivers updates via Redis Streams using small stored snapshots plus separate delta streams, with a push+pull hybrid and a local leader per petshop to smooth spikes. It explains the deliberate choice to avoid Kafka in favor of a lightweight Qx broker, node-local caches and one connection pool per machine, and reports load testing up to tens of thousands of terminals with update peaks of several thousand per second using a custom bomber for property-style tests.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 10:00Replication for thousands
- 23:46Periodic pull approach
- 39:47Kafka failure
- 50:05Redis Streams caching
- 1:00:46Testing with N.Bomber
- 1:15:02Q&A
related talks
19mBuilding a Distributed Data Ingestion System with RabbitMQ
30mBuilding domain specific databases with a distributed commit log
36mBuild a Cloud Native Asynchronous Messaging System For Scale With Redis
41mStreaming your database - Easier said than done?
43mHot Deploying Low-Latency Services for 24/7 Operation
14mA Cloud-native Architecture for Replicated Data Services
28mTobias Petry :: Scaling Databases
33mThe rise and fall of the full stack database