A Real Time Analytics Framework
summary
Yahoo builds a reusable real‑time analytics framework that surfaces events in seconds (goal: <1 minute) by combining Storm topologies for in‑memory aggregation, Kafka as the event bus and a 15‑node broker cluster with RF=3, and lightweight backing stores like LevelDB for 2–3GB lookup datasets. Benchmarks target a 6 Gbit/s firehose, ~78 Gbit/s cross‑broker traffic, micro‑batching of 100 events (adds ~100 ms) and LZ4 compression, yielding 10–15% extra compression and production‑scale throughput across multiple product teams.
AI-written summary · about
chapters (10)
- 0:00Introduction to real-time analytics
- 1:21Speaker introduction and goals
- 2:50Why build a framework at Yahoo
- 4:17Example architecture overview
- 6:04Supporting tools for consistency
- 7:50Building common libraries
- 8:31Performance requirements and challenges
- 10:01Benchmarking and optimizations
- 12:53Batching and compression techniques
- 15:22Framework journey and Q&A
related talks
08mReal-Time Data Systems: Speed, Scale, and Resilience
36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
39mReal Time Datamining and Aggregation at Scale
57mEmergence of Real-Time Analytics: Real-time Analysis of...
35mReal Time Marketing with Kafka, Storm, Cassandra & Spark #bbuzz
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns