PREVIEW: Running Kafka as a Service at Scale
summary
Covers Confluent’s experience running Apache Kafka as a multi-tenant managed service (Confluent Cloud) and the engineering changes made to support that scale. Describes concrete resilience work (replication protocol improvements, controller rewrite, reducing ZooKeeper failures), expanded observability (client, storage, replication, controller metrics for external alerting), and extensive testing and fault-injection suites used nightly to find and fix issues; also briefly mentions code-quality and compile-time bug detection practices.
AI-written summary · about
related talks
50mRunning Apache Kafka on Kubernetes and OpenShift
06mSponsored Keynote: What I Learned Running 10,000+ Kubernetes Clusters
41mAbstractions on Abstractions, Are You Building for Kubernetes?
39mLessons Learned Migrating an Existing Product to a Multi Tenant Cloud Native Environment
29mKubernetes Reloaded: AKA How to Avoid the Dev Plateau with CRDs
39mManaged CF-as-a-Service: An Operator's Cookbook
24mAstronomer's Journey from Cloud Lock-in to OSS Independence
45mMulti-tenant Flink-as-a-Service on YARN