Scale with NSQ: a realtime distributed messaging platform
summary
An open-source, Go-built messaging platform created at Bitly runs in production for over three years and handles the load behind 8 to 10 billion clicks per month, about 8,000 requests/sec. It implements topics and channels to decouple producers and consumers, uses buffered channels with a high watermark to spill to disk, and guarantees at least once delivery with possible duplicates and unordered messages. Service discovery uses nsq lookup D, and cluster ops use nsq admin; the design supports sharding, local publishes, and archival consumers.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:21How NSQ works
- 2:47Basic messaging patterns
- 9:20Lookup D
- 13:12NSQ guarantees
- 15:21NSQ Admin
