Building an Open Source Streaming Analytics Stack with Kafka and Druid
Talkat Embedded Linux Conference + OpenIoT Summit North America 2017 · on Thursday, 2017-02-23· at 14:30· Galleria South·
summary
Shows a concrete stack: Kafka as durable, partitioned event buffer with at-least-once delivery; Apache Samza as staged stream processor that shuffles partitions, writes intermediate topics, and performs a stateful join (impressions + clicks → is_clicked); and Druid as a time-partitioned, columnar query store with indexers and historicals serving sub-second ad hoc queries and approximate algorithms. Demonstrates real numbers (example: 2.8 million edits and 1.1 billion characters) and a production pattern for low-latency exploration and alerting.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 0:25Overview
- 1:06The Problem
- 2:03Events
- 3:13Example
- 6:48Problems
- 8:53Models
- 9:21Data Delivery
- 9:43Data Delivery Problems
- 12:48Kafka Summary
- 13:49Stream Processing
- 15:04Stream Processing Challenges
- 15:48Stream Processing System
- 18:02Challenges
- 19:39Subheading Queries
- 21:26Technical Overview
- 23:03Approximation Algorithms
- 24:13Druid Architecture
- 27:59Rules of Example
- 30:01Raw Data
- 30:56Shuffle
- 31:27Join
- 32:10Joint
- 33:08Conclusions
related talks
37mBuilding an Event Analytics Pipeline with Kafka, ksqlDB, and Druid
27mSucceeding with Apache Druid and Clickstream Data
32mBuilding Real-time Systems with Open Source Technologies
20mAnalyzing streaming data with Apache Druid
46mScalable Realtime Analytics using Druid
20mReal-time analytics with Flink and Druid #bbuzz
34mChasing the Stream Processing Utopia
49mLet’s Build a Streaming Data Pipeline!