Extending Flink state serialization for better performance and smaller checkpoint size
summary
Shows that Kryo fallback often causes most CPU and state bloat, demonstrates registering custom serializers (protobuf, Scala sealed traits) and replacing string and primitive-array encoders, and gives benchmarks and code on GitHub. Reports a real case where changing serializers and using variable-length encoding and delta encoding cuts state from ~100 GB to ~40 GB and improves throughput; microbenchmarks show simpler string/varint writers run up to ~2x faster. Library published on Maven (flink-adt).
AI-written summary · about
chapters (32)
- 0:00Introduction
- 2:38Stateful aggregations
- 3:53State processing
- 5:01Flame graph
- 10:06Use case
- 11:55Why
- 13:01Magic
- 13:33Micro
- 14:17Keys
- 14:46Sealed traits
- 15:23Sealed traits with Creole
- 15:57Scala compiler doesnt know
- 16:27Custom type information
- 17:04Nested types
- 17:34Micro function
- 17:47Shapeless
- 18:55Shapeless with Scala
- 19:49Demo
- 21:22Performance
- 22:06String Sterilizer
- 22:59Strings
- 23:59Variable length encoding
- 24:28Benchmark results
- 24:59The main problem
- 26:12Under footprint
- 27:03Primitives
- 29:35VAR int benchmark
- 31:12General results
- 31:40How to use
- 31:52String civilization
- 32:47GitHub
- 33:14Questions