Under-The-Hood of Druid without Zookeeper on Kubernetes
summary
Claim: Druid runs without ZooKeeper on Kubernetes by replacing coordination with Kubernetes APIs and an HTTP long‑polling state sync layer. Implementation uses HTTP long polling with Jetty async threads and a 1000-update history per node to resume watchers, node discovery via pod labels + annotations, and leader selection via Kubernetes’ lease-based leader election (resourceVersion test-and-set). The extension covers four state-sync flows plus discovery and leadership; it works by keeping broker caches updated and allowing a Druid cluster to operate without ZooKeeper.
AI-written summary · about
chapters (17)
- 0:00Intro
- 2:35Druid: Features
- 4:25Druid Architecture
- 6:01Druid: Data Ingestion
- 10:27Druid: Segment Loading with Zookeeper
- 12:39Druid: Segment Announcement with Zookeeper
- 13:56Druid: Query Processing Overview
- 14:40Druid: Zookeeper Use Cases
- 16:56Druid: State Synchronization With Zookeeper
- 19:31Druid: Segment Announcement with Http Long Polling
- 24:08Druid: Segment Change Request History
- 25:01Jetty Http Threads
- 25:39Asynchronous Request/Response
- 26:18Druid: All Zookeeper Use cases
- 27:45Druid: Pluggable Discovery and Leadership Interfaces
- 28:39Kubernetes: Pods
- 31:23Leader Election Algorithm
related talks
26mReleased From the Cage: Apache Kafka Without Its ZooKeeper
20mA ZooKeeper Layer for FoundationDB
33mManaging a Kafka cluster in KRaft mode with Strimzi
24mOxia - A Horizontally Scalable Alternative to Apache Zookeeper
22mIntroducing Zelos - Zookeeper API leveraging Delos
46mKubernetes Stateful Application Disaster Recovery with Zero Data Loss
27mService Discovery Challenges at Scale
44mDistributed Locking in Kubernetes