Co-designing Raft + thread-per-core execution model for the Kafka-API
summary
Redpanda is a Kafka‑API storage engine built around a thread‑per‑core model and a co‑designed Raft path, using the systar library, per‑core arenas with no virtual memory, an IO‑buffer iterator and group‑commit/reahead to reduce disk syncs. Claims: 10–100× better tail latencies overall and a recent two‑phase + trigger pipeline that gives >10× improvement in certain saturation cases; group commits drop disk operations ~20%. Uses preallocation (fallocate), Jepsen for correctness testing and an rpk autotune probe for hardware tuning.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:53Red Panda
- 5:13Overview
- 7:16Onon observation
- 9:18The evolution of streaming
- 12:30The new bottleneck
- 13:56Threadpercore model
- 17:50Asynchronicity
- 21:23No Virtual Memory
- 24:59Fragmented Buffer
- 28:13Memory Allocation
- 31:06Direct Memory Access
- 33:48No Page Cache
- 38:42Adaptive Allocation
- 45:34Request Pipelining
- 50:15Core Local
- 55:48Request Splitting
related talks
55mImplementing Raft in RabbitMQ
21mApache Ratis - A High Performance Raft Library
43mRa: a Raft implementation
02h 04mKafka with Raft in Production: A New Era for Distributed Commit Logs
38mRe-architecting Apache Kafka for the Cloud
37mLet's build our own message streaming platform
37mImplementing Raft in RabbitMQ
44mCluster Consensus: When Aeron Met Raft