SnailTrail: Generalizing Critical Paths for Online Analysis of Distributed Dataflows
summary
System generalizes critical path analysis to streaming jobs and reports a per-activity critical participation score computed on time-based windows from a program activity graph. It computes that score without enumerating all paths by adapting Brandes-style betweenness centrality. Instrumentation is lightweight (no observed overhead for Spark and TensorFlow, ~10% for Flink and Timely), runs online, and ingests high rates (about 1.2 million events per second on 8 workers). With 48-worker sources it processes 1 second of trace in ~6 ms and 256 s in <25 s.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:20Snail Trail: Diagnosing latency issues in dataflows "Where is the latency bottleneck in my computation?"
- 1:32Snail Trail works online with minimal instrumentation
- 2:31Example 1: Metrics in Flink's dashboard
- 3:18Example 2: Task Scheduling in Spark
- 4:52The real-world is more complex
- 5:44Conventional profiling can indicate wrong bottleneck
- 6:52The program activity graph
- 8:18Classical critical path analysis
- 10:18Program activity graph window
- 10:46Cannot enumerate all critical paths
- 11:20Sampling critical paths misses critical activities
- 12:42The Critical Participation metric
- 14:25Activity type bottleneck analysis (Spark)
- 15:19Operator bottleneck analysis (Flink)
- 16:27Snail Trail performance Low instrumentation overhead
- 17:48Summary
related talks
38mSelf-managed and automatically reconfigurable stream processing
36mData processing at Spotify - Why portability matters
24mData analytics with distributed tracing data
38mStreaming Fast and Slow
25mData: Moving with the flow
42mPrinciples and Patterns for Streaming Data Analysis
21mSplunk Data Stream Processor
34mA journey of scaling high-volume low-latency stateful processing