Scaling your Kafka streaming pipeline....
summary
Nielsen Marketing Cloud runs a multi-petabyte Kafka pipeline that ingests ~25 billion events/day (≈300k/sec, peaks to 5M/sec, ~200 TB/day) across 60 brokers and ~9,000 partitions. The rebuild replaces micro-batch consumers with isolated hourly tasks that spin up Airflow + Spark on EMR clusters, use the Kafka timestamp API to read exact offsets for time ranges, cut costs from ~$540k to ~$160k/year, speed outage recovery from 7h to ~40m, and reduce retention from 3 to 2 days.
AI-written summary · about
chapters (12)
- 0:00Introduction and Kafka overview
- 2:27Real-world Kafka outage case study
- 6:18Problems with Kafka scaling limits
- 9:46Old Spark streaming pipeline issues
- 12:35Why adding partitions isn't ideal
- 15:46New pipeline design goals
- 17:10Isolated task processing concept
- 20:41Airflow and EMR auto-scaling mechanics
- 22:47Cost and performance improvements
- 25:19Cluster utilization results
- 25:40Key architecture insights
- 27:26Q&A scaling with fixed partitions
related talks
33mScaling your Kafka pipeline can be a pain - but it doesn’t have to be
32mShould you read Kafka as a stream or in batch? Should you even care?
35mKafka Deployment at Scale
38mData Pipeline with Apache Kafka
01h 00mPutting Apache Kafka to Use
43mBuilding Scalable and Extendable Data Pipeline
39m400 Billion Messages a Day with Kafka
42mBuilding a Data Pipeline with Clojure and Kafka