Buffer-based End-to-end Request Event Monitoring in the Cloud

on YouTube ↗16m

" summary

BufScope models the request datapath as a buffer chain and injects per-request IDs plus a remaining-size field into packet headers so SmartNICs and programmable switches can link buffer events back to RPCs. Implemented on commodity SmartNICs and programmable switches, it detects five buffer-event types, diagnoses 95% of RLAs, and reports <0.07% network bandwidth overhead with <1% application throughput decline. Real deployments on Alibaba storage show root causes located across host and network layers and diagnosis time shrinks from days to minutes.

AI-written summary · about

" chapters (17)

1. 0:00Intro
2. 0:34RPC: Fundamental component in Clouds
3. 1:04RPC Latency is Crucial for App Performance
4. 1:54RPC Latency Anomaly is Difficult to Diagnose
5. 3:45Why Difficult?
6. 4:06Any Opportunity?
7. 4:27What's the Right Unifom Model for RLA Locating?
8. 6:09Buffer Chain Abstraction
9. 6:48Design Challenges
10. 7:48Event Definition based on Buffer Classification
11. 9:49Obtaining RPC Semantics in NIC and Switch
12. 10:46Per-packet RPC-level Semantic Injection
13. 11:39CPU Overhead
14. 12:05BufScope Implementation with Low Overhead
15. 12:32Case Study 1: Polling Hang in Host Thread
16. 13:30Case Study 2: Priority Contention in Recv NIC
17. 14:13Evaluation Setup

" related talks

• 12mDebugging Transient Faults in Data Centers using Synchronized Network-wide PacketRaj Joshi, Pravein Govindan Kannan, Nishant Budhdev, Mun Choon Chan · NSDI ’21
• 19mNetwork X-Ray Vision: eBPF for Cloud-Native ObservabilitySai KR Pentaparthi · Site Reliability Engineering (SRE) 2025
• 16mCredence: Augmenting Datacenter Switch Buffer Sharing with ML PredictionsStefan Schmid, Vamsi Addanki, Maciej Pacut · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)
• 20mNetwork Monitor: A Tale of ACKnowledging an Observability GapJason Gedge · SREcon19 Europe/Middle East/Africa
• 12mBreaking the Transience-Equilibrium Nexus: A New Approach to Datacenter Packet TransportMohammad Alizadeh, Anirudh Sivaraman, Shiyu Liu, Ahmad Ghalayini, Balaji Prabhakar, Mendel Rosenblum · NSDI ’21
• 39mUsing EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance IssuesDongdong Tao · Cephalocon 2023
• 17mBarre: Empowering Simplified and Versatile Programmable Congestion Control in...Zicheng Wang, Haohan Xu, Xiaoliang Wang, Zhuo Jiang, Xiaolong Zhong, Jianxi Ye, Haoran Wei, Yajuan Peng, Xiaoming Fu, Junkai Huang, Yang Bai, Huichen Dai · USENIX ATC '25
• 23mNetBouncer: Active Device and Link Failure Localization in Data Center NetworksZe Jin, Chuanxiong Guo, Cheng Tan, Tianrong Zhang, Karl Deng, Haitao Wu, Dongming Bi, Dong Xiang · NSDI 2019