dShark: A General, Easy to Program and Scalable Framework for Analyzing In-network

on YouTube ↗22m

" summary

dShark provides a MapReduce-like pipeline that parses mirrored packets into compact summaries using a JSON declarative spec and runs per-group analysis with C++ per-group UDFs, correlating instances despite middlebox-added headers. It tolerates capture noise by using TCP-seq evidence, implements 18 ready apps, and processes >10 Mpps on a single commodity server with near-linear scale to ~26.4 Mpps (16 passers, 24 groupers). Drop-localization reduces false positives from ~5,600 to 7.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 0:32Network reliability is critical
3. 0:55Existing tools are the first attempts
4. 2:17In-network packet capture is the last resort
5. 3:13In-network analysis: challenges
6. 3:49Example: Route error checker
7. 5:32Packet headers are modified by middleboxes
8. 6:55dShark: three goals
9. 7:30How operators manually process traces
10. 9:53dShark's programming model
11. 11:16Declarative spec. in parsers and groupers
12. 12:20Same protocol headers bring ambiguity
13. 13:11Imperative diagnosis logic in the query processors
14. 14:02dShark overview
15. 14:35Another example: load balancer profiler
16. 16:09Tolerate capture noise
17. 17:21Performance of Shark
18. 20:24Conclusion

" related talks

• 05mHow to Analyse a DDoS QuacklyRemco Poortinga – van Wijnen · DuckCon #4
• 20mDiamond Miner: Comprehensive Discovery of the Internet's Topology DiamondsRobert Beverly, Kevin Vermeulen, Justin P. Rohrer, Olivier Fourmaux, Timur Friedman · NSDI 2020
• 19mDisaggregating Stateful Network FunctionsRanveer Chandra, Deepak Bansal, Balakrishnan Raman, Sachin Jain, Rishabh Tewari, Neeraj Motwani, Evan Langlais, Soumya Tiwari, Rishiraj Hazarika, Deven Jagasia, Michal Zygmunt, Avijit Gupta, James Grantham, Srikanth Kandula, Stewart Grant, Gerald DeGrace, Silvano Gai, Mario Baldi, Krishna Doddapaneni, Arun Selvarajan, Arunkumar Arumugam, Pranjal Srivastava · NSDI 2023
• 17mDifferential Network AnalysisXu Liu, Peng Zhang, Aaron Gember-Jacobson, Yueshang Zuo, Yuhao Huang, Hao Li · 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22)
• 14mEndangered Privacy: Large-Scale Monitoring of Video Streaming ServicesMartin Björklund, Romaric Duvignau · USENIX Security '25
• 12mBridging Missing Gaps in Evaluating DDoS ResearchJun Li, Lumin Shi, Samuel Mergendahl, Devkishen Sisodia · 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET '20)
• 24mScalable Small Message Aggregation on Modern InterconnectsOscar Hernandez, Aaron Welch, Wendy Poole, Stephen Poole · VIVEKFEST 2024
• 26mNetwork monitoring with FOSSBenet Leong · FOSSASIA Summit 2017