Scaling your Kafka pipeline can be a pain - but it doesn’t have to be
summary
A multi‑petabyte Kafka pipeline (25B events/day ≈300k eps, 5M eps peak, 60 brokers, 9k partitions) moves from continuous microbatch Spark readers to hourly isolated Spark clusters started by Airflow reading precise offsets and landing to S3. EMR autoscaling drops wasted capacity, cutting costs from ~$400k/yr to ~$160k/yr (~$240k saved, ~60% reduction) and shortens recoveries from seven hours to ~40 minutes—an Airflow + EMR autoscaling pattern that yields 85% faster recovery in production.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:50Expiration in a Data Pipeline
- 3:58About Nielsen Marketing Cloud
- 5:42Data Structure - Topics and Partitions
- 6:57Our Old System
- 9:13Issue #1: Speed Limit
- 10:18More Wasted Money
- 12:17Recovery Time
- 12:43After an Outage
- 13:25Scaling the Pipeline
- 14:36Adding More Partitions
- 17:36Common Sense Option
- 18:12Manage & Control Consumers' Offsets
- 19:51Using Discreet Time Slots
- 21:01Benefits
- 21:30Task (& Cluster) Independence
- 23:17Comparing to Previous System
- 24:00Single Flow View
- 24:48Multi Flow View
- 26:04Cost Drop
- 27:05Outage Handling
- 29:20Process Time Single Partition
- 29:53Logarithmic Scale
- 30:26Goal - Fully Scalable Consumers
- 30:41Recap - What we've talked about
related talks
32mScaling your Kafka streaming pipeline....
32mShould you read Kafka as a stream or in batch? Should you even care?
38mData Pipeline with Apache Kafka
43mBuilding Scalable and Extendable Data Pipeline
37mScaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
01mScale By the Bay 2021 intro video by Opher Dubrovsky
24mThe Cost of Apache Kafka
26mRe-architecting the Data Platform with Spark on Kubernetes