Designing Payloads for Event-Driven Systems
summary
Practical rules for Apache Kafka payloads: use apache kafka keys and headers (route partitions, put small lookup-friendly metadata in headers), always send a top‑level object and grouped fields, keep messages under one meg and favour lightweight representations or hypermedia links over embedding files, include ISO timestamps and open telemetry trace/span ids, use Avro or Protobuf with a schema registry (Carapace), evolve schemas only backwards‑compatibly or publish a new topic (e.g. topic.v0 → topic.v1) and use go‑gen‑avro + Confluent clients for Go.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:33Event-Driven Systems
- 2:11Apache Kafka "Apache Kafka is an open-source distributed event streaming platform"
- 3:04Payloads The messages the machines send between themselves • Large string/binary data • No rules
- 4:07Apache Kafka Records Use all the features of Apache Kafka's records
- 7:05Flat or Nested Structures? • Always use a top level object structure (not an array) Group related fields together
- 8:53More Data or Less Data? • For small payloads, add the context fields • Use lightweight representation rather than the full
- 11:06Example: GitHub Webhooks (snippet from the push webhook)
- 12:54A Note on Timestamps • Apache Kafka includes publish time in the header. . Consider adding payload-level timestamps. • Timestamps only as accurate as your clock!
- 14:36Event Tracing Standards are great! • Trace ID used by every event in the story • Span ID in event, becomes Parent Span ID for child
- 18:46Schemas Schemas enforce payload structure • Avro format requires a schema
- 20:12Evolving Schemas • Aim for backwards-compatible changes
- 23:05Example: Avro Schema Avro schema example for sensor data
- 24:04GoGen makes Avro Structs gogen-avro avro machine sensor.avsc
- 24:42Go with Kafka and Avro
- 25:33AsyncAPI for Apache Kafka AsyncAPI describes event-driven architectures
- 26:00Describing Payloads The channels section of the AsyncAPI document
- 26:19Documenting Payloads
- 27:07Resources • Examples
related talks
30mDesigning Payloads for Event-Driven Systems
50mApplication Design Patterns on Apache Kafka
16mAn introduction to Apache Kafka - this is Event Streaming, not just Messaging
42mCreating event-driven microservices: the why, how and what
45mReliable Toolkit for Event-Driven Microservice Architecture
55mAvoiding mistakes with events, one event at a time
01h 00mKafka for .NET Developers
45mManage, secure, and socialize Kafka topics