AStream: Ad-hoc Shared Stream Processing
summary
AStream is a distributed ad-hoc shared stream-processing layer built on Apache Flink that claims to reuse computation across thousands of short-lived queries. It tags each tuple with a query set bitmap and encodes query lifecycle changes in a change log, slices windows into non-overlapping pieces and performs slice-based incremental join by ANDing bitmaps to skip irrelevant joins. Operators remain deterministic with exactly-once semantics. On 4– and 8-node clusters it matches Flink on single queries and gives orders-of-magnitude throughput and millisecond deployment latency as query parallelism grows.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:26Use-case
- 2:58AStream and Requirements for Ad-hoc Stream Processing
- 4:06AStream System Overview
- 4:58Data Model - Query-set
- 7:05Computation Model
- 8:47Implementation Details - Joining Slices
- 11:23Experiments
- 12:11Workload Scenario 1
- 14:06Conclusion
related talks
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems
26mHow Streaming Helps Your Staging Environment and Sandboxes Stay Up To Date
48mStreaming all the things with Akka Streams
26mHow to Pick Your Next Streaming Architecture
42mApache Beam: A Unified Model for...
32mBig Data Processing : Where do we go next ?
42mApache Flink deep-dive #bbuzz
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?