Scale By the Bay 2021 intro video by Opher Dubrovsky
summary
Concrete cases of Kafka consumer failures under rapid traffic shifts and during broker recovery, with measured numbers from a pipeline handling 250 Billion events/day. Describes Kafka consumer quirkiness like rebalancing storms, lag spikes and stale offsets and demonstrates client-side fixes: explicit backoff and retry policies, cooperative rebalancing, partition-aware scaling, and validation that these changes cut recovery time and smooth throughput on AWS with Kafka, Spark, Lambda, Airflow, OpenFAAS and Kubernetes.
AI-written summary · about
related talks
38mKafka Message Broker Under High Load
41mKafka as a Platform: The Ecosystem from the Ground Up
01h 00mIn Depth Kafka Streams
39mKafka as a Platform: The Ecosystem from the Ground Up
50mIntroduction to Apache Kafka
33mMicroservices, Events, and Breaking the Data Monolith with Kafka
37mScaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
41mKafka as a Platform: the Ecosystem from the Ground Up