Auditing Data and Answering the life long question Is it the end of the day yet?
summary
Nielsen builds Life Line: an auditing pipeline that injects an Avro audit header with a 5-minute window, emits metadata via both Kafka and an API Gateway, and consumes it into Postgres on RDS. ETL uses Spark/EMR, S3 raw storage and Superset dashboards. A Lambda API answers “is it the end of the day yet?” by checking overall arrival rate, number of windows and the last window; a sequence id detects duplication. Weekly partitions, offload to history and tuned Postgres indexes make queries fast; alerts surface region/topic failures early.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:26Nielsen's Architecture (AT THE TIME)
- 3:55Data Arrival Pain Points
- 5:43Recovering from failures
- 8:03Is it the end of day yet? Legacy answers to a legacy problem
- 12:41Auditing Window
- 15:01Auditing Header Injection
- 19:36Designing Out Output Table
- 26:25Optimizing PostgreSQL for Audit Queries
- 33:09Alerts and add-ons
related talks
37mAuditing your data and answering the question, is it the end of the day yet?
17mOptimizing Data-Driven Decisions
30mData Engineering building your BI infrastructure from scratch
29mData Acquisition: Overcoming Challenges
39mData Observability and Reliability Engineering in a Real-Time World
45mResilient Predictive Data Pipelines
33mAccidental DataOps: How we matured our organization with data, and how you can too!
49mData Platform Architecture, Evolution, and Philosophy