Deep Dive - Architecting for Million Operations Per Second!
summary
Design proves how to reach millions ops/sec by building a pipeline that generates synthetic CSVs with Dataflow, stages files in GCS, and coordinates write/read waves via PubSub and Cloud Functions, sharding load across multiple Memorystore Redis instances with consistent hashing. Tests hit about 300k writes/sec across three 64GB Redis instances and 1.2M reads/sec on replicas. Batching, parallel file processing, and pipelined Redis operations make the system meet throughput and latency goals.
AI-written summary · about
chapters (12)
- 0:00Introduction to million operations per second problem
- 1:11Customer requirements and data
- 7:12Breaking down the problem
- 12:14Storage and network needs
- 12:58Bigtable vs Redis choice
- 15:46Synthetic data for testing
- 17:32Write path to Redis
- 21:46Optimizing write throughput
- 25:19Scaling writes with clusters
- 28:28Read path for high throughput
- 31:22End-to-end test pipeline
- 33:33Key learnings and scaling
related talks
59mMastering the System Design Methodology
25mMixing data streams at scale
15mDesigning a Read-Intensive Architecture for Information Systems
32mPromoting a data driven culture in a world of microservices
49mJourney with Aerospike: Architecture Best Practices
01h 00mFrom Concept to Reality: Solving Enterprise Challenges
40mScaling Python - Bank Edition
26mArchitecting for scale