Real Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
summary
Groupon builds a realtime pipeline using Kafka-Storm-HBase-Redis that ingests and processes about 3 million updates/sec, claiming millions in extra revenue. HBase stores per-user event columns with a real-time and a batch cluster plus bulk-upload snapshots; Storm is tuned for locality and backpressure; Redis uses client-side sharding and hashes to cut memory by ~10x and handle ~3M counter updates/sec. Duplicates are mitigated with Bloom filters, and HDFS-backed Kafka topics allow replay for recovery.
AI-written summary · about
chapters (12)
- 0:00Introduction and context
- 1:39Business evolution at Groupon
- 4:41System requirements and goals
- 6:35Problems with the legacy system
- 11:13New architecture overview
- 13:52Scaling HBase and data writes
- 17:50Managing batch and real-time
- 19:23Real-time personalization
- 24:03Redis for counting updates
- 25:44Scaling and tuning performance
- 29:33Memory management and storage
- 32:03Monitoring and error recovery
related talks
35mReal Time Marketing with Kafka, Storm, Cassandra & Spark #bbuzz
41mFrom Batch to Realtime with Hadoop
53mBig Data: from mammoth to elephant
01h 00mPutting Apache Kafka to Use
17mA Real Time Analytics Framework
39mReal Time Datamining and Aggregation at Scale
30mFive Challenges in Real-time Stream Processing and Five Solutions
05mPREVIEW: 1 Day, 1 Data Hub, 100 Billion Messages