Should you read Kafka as a stream or in batch? Should you even care?
summary
A multi‑petabyte Kafka consumer is reworked from micro‑batches to hourly batch jobs using Spark on EMR, Airflow for scheduling, Kafka timestamp→offset reads, S3 for output and spot instance fleets for cost. Topic has 1,250 partitions, ~60 TB/day overall and ~20 TB/day on the delivery topic. Cost drops from $400k to $160k/year (~65% claimed savings), cluster utilization rises from ~30% to ~75–90%, and outage recovery shrinks from six hours to ~40 minutes, demonstrating isolated hourly tasks and serverless‑style scaling.
AI-written summary · about
chapters (8)
- 0:00Intro
- 2:21Stream vs Batch
- 6:20Example System
- 8:36Issues
- 12:07Efficiency
- 16:49Results
- 22:37Key insights
- 25:50QA session
related talks
33mScaling your Kafka pipeline can be a pain - but it doesn’t have to be
32mScaling your Kafka streaming pipeline....
46mKafka Streams at the Big Data Crossroads at Lightning Speed
38mData Pipeline with Apache Kafka
24mThe Cost of Apache Kafka
38mKafka Message Broker Under High Load
01h 00mPutting Apache Kafka to Use
01h 07mKafka Streams and KSQLDB