The internals of stateful stream processing in Spark Structured Streaming
summary
Covers state management in Spark Structured Streaming 2.4.3 with concrete APIs and demos: use of groupBy/rollup/cube, flatMapGroupsWithState for arbitrary stateful processing, GroupState/GroupStateTimeout, watermarks and output modes, and triggers (micro-batch vs continuous). Demonstrates MemoryStream demo with checkpointing and shows state stored as files on HDFS that can be inspected and even modified, notes default shuffle partitions = 200 (changed to 1 for demo), explains how watermarks and windowing bound state size and when state expires.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:03Overview
- 3:07About me
- 5:46Stack Overflow
- 8:19Spark Structured Streaming
- 13:20Stateful Stream Processing
- 21:20Swimming Aggregation
- 23:29Optimizing Queries
- 31:31Checkpoint Location
- 45:11Example
related talks
40mDeep dive and best practices of Spark streaming
45mData Stream Processing Concepts and Implementations
45mTracking and Triggering Pattern with Spark Stateful Streaming
01h 01mApache Spark Structured Stream: A Near-Real-Time Data Grinder
44mReal-time Data Processing with Apache Spark Structured Streaming
01h 00mIn Depth Kafka Streams
31mBlueprints for common stream processing use cases with Flink
01h 14mPANEL - Streaming.