Stop Breaking Production - Schema Evolution Demystified
summary
A concrete checklist for avoiding runtime breakage: a product-update Kafka example shows removing categories for tags stops order processing and breaks an ML pipeline, so guarantee consumer-first vs producer-first deployment order and use a schema registry (Confluent) with CI checks against backward/forward/full compatibility. Compare JSON, Avro, Protobuf: use Avro aliases and default enum values, generate code with AvroHugger/ScalaPB, and for Protobuf reserve protobuf field numbers; handle failures with dead-letter topics or stop-consumer semantics and back up the registry DB.
AI-written summary · about
chapters (12)
- 0:00Introduction and speaker intro
- 1:48Problem with schema evolution
- 3:12What is schema evolution
- 5:18Compatibility types explained
- 7:51Serialization formats overview
- 11:51Allowed schema changes
- 17:09Handling incompatible changes
- 18:33Code generation strategies
- 21:45Schema registry use
- 22:47Recommendations and best practices
- 25:17Alternative solutions
- 26:23Closing and Q&A
