The Incredible Efficiency of Log Unification!
summary
Describes unifying logs and events to enable compression, causal search and “time travel” for debugging. Presents a concrete implementation using Scala, Kafka and Jaeger, with storage options like Kafka or Cassandra and a synchronous+asynchronous trace propagation technique to preserve causality across sync→async boundaries. Shows a prototype called zoom dézoom that links traces and events in a complex advertising pipeline, deployed in production to reduce data duplication and ease diagnosis.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:16The data engineer's craft
- 4:36Level 1: It's not that much harder!
- 22:43A trace is a call tree
- 39:11Principle 2: Log unification
- 45:10Advantages of the technique
