In support of workload-aware streaming state management
Talkat 12th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage '20) · 2020-07-13..07-14·
summary
Examines how LSM-tree-based key-value stores used by distributed stream processors add avoidable cost for streaming workloads and argues for configurable backends tailored to operator state. Describes inferring or learning operator state types, sizes, and access patterns (at compile time or during execution) and shows that workload-aware state management can cut p99 latency by ~10× and double throughput.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:25STREAMING DATAFLOWS
- 1:28LARGER-THAN-MEMORY STATE MANAGEMENT
- 2:09STATE REQUIREMENTS VARY ACROSS OPERATORS
- 2:37CURRENT PRACTICE: MONOLITHIC STATE MANAGEMENT
- 3:03FLAWS OF MONOLITHIC STATE MANAGEMENT
- 4:25UNNECESSARY KEY-VALUE STORE FEATURES
- 5:20WORKLOAD-AWARE STREAMING STATE MANAGEMENT
- 5:47A FLEXIBLE TESTBED FOR STREAMING STATE MANAGEMENT - Implemented in Rust
- 6:28EVALUATION GOALS
- 6:55EFFECT OF DATA LAYOUT ON WINDOW EVALUATION
- 10:51MONOLITHIC VS WORKLOAD-AWARE STATE MANAGEMENT
- 13:49OPEN QUESTIONS
- 15:20SUMMARY
related talks
29mFully Automatic Stream Management for Multi-Streamed SSDs Using Program Contexts
29mManaging State in Apache Flink
42mDistributed Real Time Stream Processing: Why and How
40mThe Seven Deadly Sins of Streaming
34mWhat We Talk About When We Talk About Disk IO
43mThe Stream Processor as a Database
04h 05mHands-on workshop going from zero to a working LSM Tree
48mUnlocking Real-time Insights: Enhancing Your Databases With Stream Processing