Pushing Your Streaming Platform to the Limit
summary
Shows how to reproduce real streaming limits using the open source open messaging benchmark (OM) deployed via Helm on Kubernetes with a driver that assigns tasks to scalable worker pods, workload files that set message size and producer/consumer configs for Kafka and Pulsar, and a Python script that saves results and draws charts. Claims the USE method finds 80% of issues with 5% effort and recommends aiming for peak traffic / n‑minus‑one to expose bottlenecks, tune latency, and stress monitoring.
AI-written summary · about
chapters (13)
- 0:00intro
- 0:22preamble
- 0:44elad leev
- 1:14about dojo
- 4:25know your limits
- 5:41the key criteria for a benchmark
- 10:51what to look for?
- 11:09the use method
- 13:04how to benchmark
- 14:18openmessaging initiative
- 18:34workload files
- 19:40benchmark insights
- 23:45thank you for your time!
related talks
38mStreaming Fast and Slow
30mBuilding Observable Streaming Systems with OpenTelemetry
41mPushing RabbitMQ to the limit
43mRelease the locusts - distributed performance tests
19mTesting things that don't want to be tested: A game streaming story
59mGame of Streams: How to Tame & Get the Most from your Messaging Platforms
37mLet's build our own message streaming platform
01h 00mSystems @Scale Summer Q&A Week 4