An architecture for big data: Supporting journalism through transparency in advertising
summary
Index Exchange runs about 140 billion auctions/day producing ~100 TB of telemetry and ~20 GB of compressed reporting, and uses Vertica COPY from HDFS to ingest hourly aggregates. The on‑prem production cluster is ~120-node prod cluster (480 TB disk, 2,400 cores, 15+ TB RAM, RAID5); Liquibase manages schema migrations, LDAP provides SQL access, and client audit logs plus a visualization/reporting layer deliver receipts and scheduled exports (S3/SFTP/email). Planned migration adds Kafka, Kubernetes/Spark streaming at the edge and Ceph for object storage.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:15Why Index
- 2:11How programmatic advertising works
- 3:37The scale of programmatic advertising
- 4:32Data and scale
- 5:27Agenda
- 5:50Architecture
- 7:57Product needs
- 8:39Vertica architecture
- 9:30How did we choose Vertica
- 11:02Cluster overview
- 11:43Vertical loads
- 12:04Builtin copy
- 13:18Application loader
- 14:34Liquidbase
- 15:57Other considerations
- 16:34Nextgen architecture
- 19:22Wrap up
- 21:00Realtime analytics
- 22:37Kafka connectors
- 23:30Planning phase
related talks
43mArchitecting for Speed & Scale
50mOur experience at Etsy com with building “Big Data” analytic capabilitie
34mData Engineering Architecture
39mBuilding Next-Generation Audience Targeting and Analytics at NBCUniversal
01h 07mBig_Data_Meets_Digital_Advertising
49mData Platform Architecture, Evolution, and Philosophy
21mReporting Systems Built on Top of Hadoop & Lucene
27mScalable by Design: How to Build an Elastic Cloud Infrastructure...