A Kafkaesque Series of Events
summary
Investigation shows a Kafka consumer architecture with subpartition queuing and an in-memory commit buffer that still suffers a concurrency failure when one subscription dominates traffic. A single customer links ~5 million devices to one email record, producing tens of millions of lagged messages and leaving CPU near zero while most postgres connections idle. Fixes: drop/skippable updates for that subscription, change global to per-partition caps, add limits on linked records, and add centralized logs and OpenTelemetry tracing (Honeycomb) to find root cause.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:35Once upon a time
- 3:53Kafka
- 8:55Issues
- 10:40Subpartition processing
- 15:02Concession
- 16:12Review
- 18:04Goals
- 22:13Observability
- 27:27OneSignal
- 30:14Why is that a problem?
- 35:37Q&A
- 42:02Outro
related talks
27mTrivially bug free code for a greener planet
43mBlazingly Fast Message Queue on Postgres with Rust
46mScaling Pinterest
42mAspects of Continuous Deployment – A Multi-Dimensional View
30mDay2 02 Sean Klein
27mLessons Learned From Architecting the Highest-scale Operational Systems in the World
17mZero-Copy Chaos. When everything you knew about tokio...
30mHow We Engineered Value in Times of COVID?