State Unlocked
summary
Describes concrete state tools and migration guarantees: Avro schema evolution is supported (1.8) with reader/writer schema compatibility checks, Flink 1.9 adds POJO evolution, and planned 1.10 targets tuples/case classes. Introduces a serializer snapshot format to validate compatibility, a State Processor API (Savepoint Connector) that reads/writes savepoints, iterates keyed/operator state, bootstraps new operators from historical data, and enables offline fixes and safepoint rewriting; FLIP-41 aims for a unified savepoint format.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:04Anatomy of a Flink stream job upgrade
- 3:25State registration with built-in serialization
- 3:52Aside: Type Information
- 6:17Schema Evolution Status
- 7:19Challenges to schema evolution
- 9:09TypeSerializerSnapshot
- 10:22Lets Implement Avro!
- 11:09Avro Snapshot
- 14:34What's it good for?
- 16:12A relational model to state
- 18:14Reading Keyed State
- 20:23Operator Reading State
- 20:42Writing State
- 23:34State Processor API
- 25:04Unify Binary format for keyed State
- 25:36Upgradability Dry Runs
related talks
23mSchema-First Application Telemetry
29mSchemas: The Key To Data Happiness
05mModeling Your Way to Consistency
46mState of Scala
29mNew Features in Apache Spark 4.0
38mInteractive Session: Flink on Autopilot with Splunk
49mBuilding Safe, More Testable Scala Production Systems
13mSystems @Scale Summer Live Q&A Session #3