Auditing your data and answering the life long question, is it the end of the day yet?
summary
Nielsen builds an auditing subsystem called Life Line that instruments the producer to emit five‑minute metadata windows and injects an AVRO Audit header into messages, sending counts both to a Kafka audit topic and to AWS API Gateway → Lambda. Spark/EMR consumers and an RDS PostgreSQL table store counts and sequence_id values so Superset dashboards and alerts detect duplications (arrival rates >100%), latency, and missing windows. Queries get sped up with weekly partitions, indexes and parallel workers. The system reliably answers is it the end of the day yet?
AI-written summary · about
chapters (35)
- 0:00Intro
- 0:35Welcome
- 0:57Agenda
- 1:35Who am I
- 2:01Nielsens architecture
- 3:57What could go wrong
- 4:12Replication problems
- 5:35What happens after data arrives
- 6:17How do we recover
- 6:54How much do we know
- 7:23The end of the day question
- 8:31The solution
- 9:53The problem
- 10:20Application layer
- 10:44Architecture
- 11:12Design solution
- 13:11Metadata object
- 14:32Audit header
- 15:41Avra code
- 16:49Amazon API Gateway
- 18:07Kafka
- 18:31Data consumption
- 20:28Data structure
- 20:56Questions to answer
- 23:26An anonymized audit table
- 25:28Apache Superset
- 25:45Latency
- 26:34Charts
- 26:44SQL Query
- 27:08Superset Logo
- 27:24Shoutout
- 27:46How to optimize your queries
- 31:01Pseudocode
- 33:17Alerts and addons
- 36:01Questions
related talks
37mAuditing your data and answering the question, is it the end of the day yet?
39mData Observability and Reliability Engineering in a Real-Time World
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
12mObservability in Privacy Infrastructure
27mThe Sanity Check: Observability for AI Data Pipelines
38mEventually, time will kill your data pipeline
14mLogging Apache Spark - How we made it easy
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah