Tackling Kafka, with a Small Team
summary
Robinhood runs Kafka as a critical pipeline after a one-engineer team grows usage; topics hit replication factor 3, clusters ingest ~80 TB/week, and ad hoc ops break down. The team builds zero-touch broker replacement (predictable broker IDs on ASG), automates rolling restarts with Prometheus checks using Fabric, and measures success by safely managing multiple clusters and more frequent config changes. Next work is a shared Python/Go client library with built-in observability and integration/regression tests to avoid client-side chaos.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:32Who am I
- 1:03What I do
- 1:32Setting the stage
- 2:46Why Kafka
- 2:58What we needed
- 3:44Jepson analysis
- 4:34Life comes at you fast
- 5:06Executives want to make realtime business decisions
- 5:33Product managers want to run experiments
- 5:55Application engineers want visibility
- 6:28Is Kafka down
- 6:59Application Engineers Perspective
- 8:37Suffering from Success
- 9:10The Problem with the Kind of Spiderman
- 9:32No Dashboard
- 10:24Library Ownership
- 11:20Engineering Tradeoffs
- 12:42Replacing Unhealthy Brokers
- 12:59Replacing Unhealthy Hosts
- 16:30Rolling Restarts
- 17:40Technical Debt
- 22:12Overhaul the Library
- 24:02Python Ghost Shop
- 26:59Conclusion
related talks
29mKafka on Kubernetes: Does it really have to be the hard way?
35mKafka Deployment at Scale
04mPREVIEW: Running Kafka for Maximum Pain
26mProduction Grade Kafka on K8s
40mKafka, DevOps...And Resilience for All
36mKafka Without Disks: Making Streaming Brokers Disposable on K8S
13mScaling Kafka on Kubernetes
23mWhen Kafka Goes Cloud Native: Observability That Actually Works!