Scotty: Efficient Window Aggregation with General Stream Slicing
summary
Scotty implements general stream slicing to run window aggregation by adaptively choosing incremental strategies based on concrete workload properties (associative/invertible/distributive, window type, time measure, stream disorder). It keeps only slice aggregates or tuples when needed, supports sessions and sliding/tumbling windows, and scales to thousands of concurrent windows with lower memory and predictable state writes. Benchmarks show up to one order of magnitude speedup over Flink’s default operator; the code is an open-source library with Flink, Beam, and Storm connectors.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:14Aggregations in Stream Processing Pipelines
- 0:58Motivation
- 2:23Research Background
- 3:33Stream Slicing Example
- 6:08General Stream Slicing
- 9:18Impact of Workload Characteristics
- 13:04Key-Facts
- 15:00Scotty Core
- 16:45Benchmark
- 18:05Using Scotty on Flink
- 19:13Scotty Window Processor
related talks
35mScotty: Efficient Window Aggregation for your Stream Processing System
19mDisco: Efficient Distributed Window Aggregation
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
34mGoing deep with Spark Streaming #bbuzz
15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs
38mFaster and Furiouser: Flink Drift
26mHow to Pick Your Next Streaming Architecture
42mApache Beam: A Unified Model for...