Moving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
summary
Built a live sports analytics pipeline for pro cycling that shifts heavy work to read time using ClickHouse, Apache Kafka/Flink for streaming, and Lambdas/DynamoDB for fan-facing delivery. Real claim: on a tiny dev cluster with ~2 GB of race data the system serves 113,000 queries per minute (QPS ~18,891) with p90 70 ms, enabling on‑read aggregation instead of brittle precomputation. Early design fails (Step Functions message limits) force rework; final architecture targets 25,000 concurrent users with sub‑200 ms responses and 3–20 second freshness.
AI-written summary · about
chapters (17)
- 0:00Real-time OLAP journey
- 2:36OLAP latency challenge
- 4:24Why real-time OLAP matters
- 5:49Business cost of latency
- 7:56Human perception of freshness
- 9:01First principle thinking
- 10:47Guided by access patterns
- 11:27Cycling media platform use case
- 13:10Early platform deployment
- 16:01Scalability failure and pivot
- 17:29Redefining segments
- 19:16Testing ClickHouse performance
- 21:44Fan-driven story hypothesis
- 22:49Defining access patterns
- 23:52Why ClickHouse scales
- 26:21Closing arguments
- 26:43Summary and call to action
related talks
08mReal-Time Data Systems: Speed, Scale, and Resilience
47mBuilding a Real-Time Analytics Database
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
34mReal time OLAP and Stream Processing Friends or Foes
41mEnhancing Event Analysis at Scale: Leveraging Tracking Data in Sports
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
11mBig Data Analytics: Immutability & Idempotence