Quest to Connect Teams with Kafka
summary
Wealthsimple shifts a growing microservice fleet onto Kafka using Confluent managed clusters, adds a schema registry, config-as-code YAML for topics, and Backstage pages, and builds playbooks, templated Datadog dashboards and a dead‑letter retry topic. Operations pain—nighttime coordinated migrations, two-person debugging bottleneck—gives way to a Kafka working group, workshops and a gamified incident trainer. Result: about 100,000 messages a minute and developer ramp time drops—reduced consumer bootstrap from one week to one day—with monitoring, audits and standardized client configs preventing repeat outages.
AI-written summary · about
chapters (11)
- 0:00Introduction to Kafka journey
- 2:35Early Kafka adoption at 12 Simple
- 5:45Scaling challenges with growth
- 7:12Incidents and operational pain
- 8:59Documenting Kafka practices
- 11:48Building Kafka community
- 13:32Reducing incident response burden
- 14:16Standardizing Kafka tools
- 15:41Current Kafka scale and usage
- 17:27Q&A session begins
- 37:33Closing remarks
related talks
23mSimplifying Kafka Governance for Developers
23mSharing the wealth, socialising Kafka event sources in an organisation
52mWeek 3 Live Q&A
19mPains and gains of Kubernetes for Developers, story from experience
39mHow We're Democratizing Stream Processing at Zillow
13mHow simplifying software can save your engineering teams' time
28mDesigning Payloads for Event-Driven Systems
28mStructure to Streams: Applying Full-Lifecycle API Management to ED Systems