State Unlocked
summary
Flink now guarantees evolvable state schema with Avro support since 1.8 and POJO migration since 1.9, letting you add fields (defaults become null) but forbidding renames or type changes; heap state uses lazy serialization/deserialization while RocksDB does eager serialize on write and lazy migrate-on-read. The state processor API is a batch tool to read, rewrite or bootstrap savepoints (rewrite keys, change max parallelism, correct bad state, convert list→map) and can produce new savepoints for jobs to restore. Roadmap items include a unified savepoint format and a dry-run upgrade check.
AI-written summary · about
chapters (19)
- 0:00Intro
- 2:39Pop Quiz!
- 4:16Anatomy of a Flink stream job upgrade
- 6:03State registration with built-in serialization
- 8:02Evolving state schema for Apache Avro types
- 9:10Evolving state schemas for Java POJO types
- 10:01Feature set of Schema Evolution
- 11:52State Schema and Evolution
- 13:59State Serialization for Heap Backends
- 16:51State Serialization for Out-of-Core Backends
- 19:24TypeSerializer Snapshot
- 21:38Limitations to Native Schema Migration
- 23:52Use Cases
- 27:28Relational Model
- 29:06Reading Operator State
- 29:16Writing State
- 29:46Reading Keyed State
- 31:33Unified Binary Format For Keyed State
- 31:58Upgradability Dry Runs
related talks
32mStoring State Forever: Why It Can Be Good For Your Analytics
37mThe Future of Apache Flink
38mApache Flink: A Very Quick Guide
39mChangelog Stream Processing with Apache Flink
28mA look at Flink’s internal data structures & algorithms for efficient checkpointing
28mMigration of a realtime stats product from Storm to Flink
27mData enrichment patterns with Apache Flink
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?