Let's build our own message streaming platform
summary
A practical recipe for building a Kafka-like system: Iggy.rs (Apache incubator) targets 15 GB/s reads and already hits ~5 GB/s sustained, uses partition-per-stream sharding, batching to avoid per-append fsync, and eventual clustering (moved from Raft toward viewstamped-style replication). Key wins come from zero-copy serialization (≈50% faster serialize, 2× deserial, 10× tail-latency p3/4), kernel-bypass IO via IO_uring + monoio/glommio, direct I/O for durability, and a thread-per-core partition pinning model.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 4:19Building streaming platforms
- 9:13Durability and batching
- 14:05Performance challenges
- 18:57Reducing latency with IO
- 25:42Alternatives to fsync
- 31:40Runtime and thread architecture
- 35:37Benchmarking and tuning
- 36:21Closing
related talks
50mPanel discussion: The new RabbitMQ Streams
30mEvent Streaming and the Future of Applications
18m"Kafka's Metamorphosis"
28mConversational big data
34mChasing the Stream Processing Utopia
28mBuild your next-generation stream platform based on Apache Pulsar
50mIntroducing Stream Support in RabbitMQ
53mDemystifying Kafka