Deep dive and best practices of Spark streaming
summary
Explains Spark Streaming internals and practical fixes: receivers buffer data on executors, direct Kafka avoids receivers so scale by increasing Kafka partitions, and one-second batches with a 30-minute window create 1,800 batches that overwhelm processing. Recommends mapWithState vs updateStateByKey because mapWithState performance scales with batch size not total state, use checkpointing for data and checkpoint metadata for driver recovery, prefer multiple receivers or repartition to spread load, and use transform to join streams with static RDDs; structured streaming is the next step.
AI-written summary · about
chapters (7)
- 0:00Introduction to Spark Streaming
- 1:25Spark Streaming lifecycle
- 9:24Aggregations and windowing
- 14:00State management best practices
- 21:56Operationalization tips
- 31:02Key benefits of Spark Streaming
- 34:51Q&A
related talks
48mThe internals of stateful stream processing in Spark Structured Streaming
27mNavigating Streaming Infrastructure
45mTracking and Triggering Pattern with Spark Stateful Streaming
44mData Streaming: Which horse to bet on?
45mData Stream Processing Concepts and Implementations
39mEvaluating Streaming Framework Performance for a Large-Scale...
01h 00mIn Depth Kafka Streams
21mAvoiding Spark Pitfalls at Scale