Step By Step Guide to Building an Application Analytics System
summary
An internal analytics pipeline called WAM ingests events from ~0.5 billion users (roughly 0.5 trillion events/day, ~5 TB/day before compression), stores records in a sparse long format, compresses rotated logs with lz4 (observed ~6:1), and exposes a SQL-like DSL that compiles to C and runs in parallel (GNU parallel) over local files. Schema lives in JSON in git so there is no ETL, histograms and sampling are built in, distinct counts use hyperloglog, and single 32-core nodes reach ~6–10 GB/s.
AI-written summary · about
chapters (27)
- 0:00Start
- 0:30Motivation: mobile application analytics
- 1:20Wam: WhatsApp Application Metrics
- 10:14You can do it too!
- 11:08Application analytics?
- 15:53Data model: example 1
- 17:20Data model: schema
- 18:31Schema management
- 19:06Data model: relational interpretation
- 19:36Storage format: traditional approaches
- 23:12Storage format: wam approach (with RLE)
- 23:57Storage: compression
- 24:28Storage: basic write cycle
- 25:11Query language: example
- 25:32Query language: motivation
- 26:04Query system: implementation
- 26:45Query compiler
- 27:39Query system: runtime
- 30:04API (protocol)
- 30:46DB shell
- 31:22Local storage: idea 1
- 32:48Local storage: use filesystem as index
- 33:13Local storage: application-level indexing
- 33:38Local storage: tables
- 33:51Distributed storage: simple approach
- 36:27References
- 37:20Data model: common dimensions
related talks
11mBig Data Analytics: Immutability & Idempotence
29mWide Event Analytics
45mBuilding an Analytics Platform
55mCloud Analytics: from 0 to 100,000 events per second in 50'
49mTransforming Analytical Data with Event-Driven Architecture
24mGive meaning to 100 billion analytics events a day, analytics at Teads
44mBest practices for analytics and generative AI on AWS(ANT302)
37mMeasuring 2.0 – Building the next generation webanalytics ...