Confluo: Distributed Monitoring and Diagnosis Stack for High-speed Networks
summary
Confluo claims an end-host stack that captures packet-header telemetry at line rate on 10Gbps links using a single core by storing fixed-size header fields in an append-only Atomic MultiLog, relaxing full transactions to linearizability and indexing with a perfect carry tree. Evaluation shows millions of writes per second (≈25M pps with one filter), thousands of per-packet triggers and tens of filters at line rate for ≥128B packets, near-linear scaling across cores, and a coordinator that produces linearizable distributed snapshots (but currently ignores cross-host ordering delays).
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:22Motivation
- 0:47Opportunity: Networks can capture a lot of data...
- 1:32Example: Checking Path Conformance
- 1:57Goals for end-host stack design
- 2:27Challenge: ... Networks capture a lot of data
- 3:19Existing Approaches
- 6:07Atomic MultiLog: Write Efficient Storage
- 7:04Atomic MultiLog Consistency
- 8:12Efficient Linearizablity for Logs
- 9:18Atomic MultiLog Linearizability
- 10:52Atomic MultiLog Indexing
- 13:04Confluo End-host Architecture
- 14:30Consistency in Distributed Diagnosis
- 15:27Evaluation
- 16:09Atomic MultiLog Performance
- 18:34General Applicability and Impact
- 19:34Confluo Summary
related talks
23mBuilding a network stack for optimal throughput / low-latency trade-offs
18mMeasuring Congestion in High Performance Datacenter Interconnects
12mConcordia: Distributed Shared Memory with In-Network Cache Coherence
19mSwCC: Software-Programmable and Per-Packet Congestion Control in RDMA Engine
30mNoisy Neighbors, through Networking
16mState-Compute Replication: Parallelizing High-Speed Stateful Packet Processing
16mFLB: Fine-grained Load Balancing for Lossless Datacenter Networks
14mHigh-throughput and Flexible Host Networking for Accelerated Computing