Processing Streaming Data at a Large Scale with Kafka
summary
Shows how a Rails app uses two Kafka topics and three Rake tasks to turn raw nginx logs into aggregated counts, with an importer that fakes streaming, a preprocessor that GeoIPs IPs and sets a partition key like city.country, and an aggregator that does five-second batching into a single ActiveRecord table. Kafka handles partitioning, replication across brokers and consumer groups with partition reassignment so rebalancing happens within a minute and DB writes drop from per‑line updates to periodic bulk increments.
AI-written summary · about
chapters (7)
- 0:00Start
- 1:40Streaming Data
- 8:57Explain Kafka
- 13:48Consumer
- 21:27Step Two
- 26:30Demo
- 30:21The Format of the Messages
related talks
31mStreaming: Real-Time Architecture for Connected Products in Automotive
45mStream Processing with KSQLDB and REST API Queries
47mStream All the Things!
23mDeliver real-time data to customer using Streaming APIs
08mGetting Real with NodeJS and Kafka: An Event Stream Tale
21mREST APIs vs. Data Streams: Which Is Right for Your Business?
29mRealtime web applications with streaming REST
30mBig Data and Small Data Everywhere, All at Once