Fast ACS: Low-Latency File-Based Ordered Message Delivery at Scale
summary
Fast ACS is a file-based ordered delivery system that combines file-based ordered delivery, memory caching (Clickmap L1) and Colossus L2, chunked files across brokers, and RDMA reads (Pony Express) so consumers fetch bytes directly. Deployed to dozens of clusters, it scales to tens of thousands of consumers per cluster (experiments show ~8,000 consumers at 70 Gbps and ~20,000 reaching ~1.8 Tbps), adds ~25k LOC, achieves p99 ≈ 730 ms in production and 99.9% cache availability with Colossus fallback.
AI-written summary · about
chapters (9)
- 0:00Problem overview
- 1:26Traditional system limitations
- 2:36Proposed technical solutions
- 4:26System architecture
- 6:16Cache layer and flow
- 8:25Cross-cluster replication
- 11:02Consumer read strategies
- 12:23Evaluation and experiments
- 14:41Production experience
related talks
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
32mCDN Scaling Challenges
16mAchieving Wire-Latency Storage Systems by Exploiting Hardware ACKs
22mAsynchronous Computing at Scale: Overview and Learning
15mACCL+: an FPGA-Based Collective Engine for Distributed Applications
38mHazelcast That Could: Getting Lightning-Fast Responses at Hundreds of RPS
22mEveryone Loves File: File Storage Service (FSS) in Oracle Cloud Infrastructure
40mOpen Source Memory Speed Virtual Distributed Storage System