Batching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
summary
Claims a production pipeline that scales from 1 to 100+ million users in a single month by replacing ETL/Athena batch jobs with a continuous analytics stack: Kinesis ingestion, an Apache Flink application that joins beacons into page impressions, resolves user agents, writes raw PIs to S3 and one-minute aggregates (histogram buckets) to Elasticsearch for Kibana dashboards. Split tests with speed kit show a clear left-shift in load-time histograms; end-to-end latency drops from ~90 minutes to sub-minute reporting and the dashboards remain fast.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:33You Heard The Stories
- 2:353 Things Make Your Website Slow
- 4:02Speed Kit Makes Websites Fast
- 5:12Measuring the Uplift - With SCIENCE
- 6:16Real-User Monitoring (RUM)
- 7:11Our RUM Pipeline
- 8:11When to Send Data Beacons ?
- 9:30Types of Data Beacons
- 11:21Schema: Sessions
- 12:08So Where is the Problem?
- 12:29Schema Overview
- 14:02Our Batch Analytics Tech Stack
- 16:24Processing Stages & Latency
- 17:02Early 2020: AWS Prototyping
- 19:32Shiny & Newold & Lame Schema
- 20:473 Levels of Aggregation
- 23:01Kibana Dashboard: Performance Uplift
- 25:20Speed Kit: Web Performance Plugin
- 25:35Split Testing for Web Performance
related talks
22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
47mBuilding Near Real-time Analytics Solution on AWS
11mData science: Next-gen performance analytics
27mArchitecting Sub-Second Analytics Pipelines
37mMeasuring 2.0 – Building the next generation webanalytics ...
39mBig Data Analytics With AWS Athena
26mHow We Prepped Our Infrastructure for a Super-Bowl Sized Traffic
29mA New Kind of Analytics: Actionable Performance Analysis