Managing State in Apache Flink

on YouTube ↗29m

" summary

Shows how Flink stores operator state as state snapshots to distributed filesystem using checkpoint barriers, and contrasts JVM-heap vs eager vs lazy serialization with RocksDB doing on-access (lazy) deserialization. Recommends avoiding Kryo for persistent state and using Avro/Thrift or custom serializers, explains how to register state early (open lifecycle) instead of lazily, and documents serializer upgrade hooks that allow restoring savepoints only when the new serializer is compatible.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 1:09Recap
3. 11:02State Declaration
4. 17:23State Serialization
5. 21:19State Migration

" related talks

• 28mRundown of Flink's Checkpointsdavid visikovic · Flink Forward Global 2021
• 42mApache Flink Worst PracticesKonstantin Knauf · Flink Forward Europe 2019
• 42mInterview and AMA with the creators of Stateful FunctionsSeth Wiesman, Stephan Ewen, Robert Metzger, Igal Shilman · Flink Forward Global Virtual 2020
• 42mApache Flink Worst PracticesKonstantin Knauf · Flink Forward Virtual 2020
• 24mStateful Functions: Flexible like a Microservice, Consistent like a MonolithSeth Wiesman · Flink Forward Global 2021
• 51mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Voxxed Days Ticino 2025
• 38mDeploying Stateful FaaS on Streaming DataflowsMarios Fragkoulis, Adil Akhter · Flink Forward Europe 2019
• 38mPolyglot Actor Model on Flink Stateful FunctionsTimothy Bess · Flink Forward Global 2021