Auditing your data and answering the question, is it the end of the day yet?
summary
Nielsen builds a data-auditing pipeline that injects an audit header into Avro messages at the producer, ships audit-window metadata via an AWS API Gateway, stores events on S3 (Parquet) and writes counts into PostgreSQL RDS. A configurable five-minute window aggregates per topic/server/process; arrival-rate checks, expected window counts and the last-window rate together decide “end of day.” Tools: Kafka + Schema Registry, Spark, Airflow partitions, Superset dashboards; duplication detection uses a running sequence id, and alerts surface regional replicator stalls (seen as >100%, sometimes >160–200% arrival rates).
AI-written summary · about
chapters (24)
- 0:00Intro
- 2:02Nielsen's Architecture (AT THE TIME)
- 4:49Data Lake
- 5:37Data Arrival Pain Points
- 7:31Recovering from failures
- 8:54Is it the end of the day yet? When do we process data?
- 9:59Is it the end of day yet? Legacy answers to a legacy problem
- 11:57Little Fires Everywhere
- 12:25Auditing window? Let's design our metadata
- 17:04Auditing Header Injection
- 18:44Shipping Audit Window to Collection Point
- 19:37Consuming Audit Data
- 20:25In Context
- 21:00Storing Data and Querying to Optimum
- 21:52Designing Out Output Table
- 26:52Shout out to my dad....
- 27:13Optimizing PostgreSQL for Audit Queries
- 28:33Managing Partitions with Apache Airflow
- 29:25Offloading Data to History
- 32:36Scheduling your spark job
- 33:13It is not the end of the day
- 33:54Alerts and add-ons
- 34:15Alerting system
- 35:23Detecting duplications
related talks
39mAuditing your data and answering the life long question, is it the end of the day yet?
37mAuditing Data and Answering the life long question Is it the end of the day yet?
17mOptimizing Data-Driven Decisions
26mA Journey of Auditlogs from Kafka to ElasticSearch
29mData Acquisition: Overcoming Challenges
22mEnd-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline
01h 01mDealing With Data Panel
32mBuilding a Real-time Ingestion Platform using Kafka and Aerospike