Calculating the Value of Pie: Real-Time Survey Analysis With Apache Kafka®

on YouTube ↗01h 00m

" summary

Builds a Telegram bot that writes survey entries (Avro, schema registry) into Kafka, uses ksqlDB to create streams and a histogram aggregation, and outputs both real‑time and a 48‑hour tumbling window final result. Uses compact topics for respondents, emits changes vs emit final to control updates, and pushes the final verdict back to the phone via a Kafka Connect HTTP sink. Pipeline runs on Confluent Cloud, produces live counts and a final shopping list — it works (pumpkin and apple pies win).

AI-written summary · about

" chapters (32)

1. 0:00Introduction
2. 1:20What are you baking
3. 4:41The problem statement
4. 5:50Surveys are everywhere
5. 7:59Survey analysis in general
6. 9:12Making it more exciting
7. 10:22What is Apache Kafka
8. 13:19Event Platform
9. 14:40Pie Example
10. 18:12Kafka Architecture
11. 23:09Kafka Cluster Example
12. 27:07Where to Start
13. 28:27Understanding the Data
14. 29:24Kafka Schema Registry
15. 33:02Kafka Survey IDs
16. 34:24Write Survey to Kafka
17. 36:32Conversation Handler
18. 37:14Message Handler
19. 38:14Send
20. 40:55Processing Data with Kafka
21. 44:46Define Inputs
22. 46:19Normalize Data
23. 48:53Survey Responses
24. 49:12Create Table
25. 50:17Create Histogram
26. 51:05Output
27. 51:24RealTime Results
28. 51:53Late Results
29. 52:56Kafka Connect
30. 55:18Summary
31. 56:26Baking something
32. 58:46Stream processing use cases

" related talks

• 59m🤖 Building a Telegram bot with Apache Kafka and ksqlDBRobin Moffatt · NDC London 2021
• 41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · All Things Open 2020
• 29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data PipelineRobin Moffatt · DevOps Pro Europe 2021
• 31mFun with SensorsYogi Rampuria · Distributed SQL Summit Asia 2022
• 57mTaming IoT Data: Making Sense of Sensors with SQL StreamingHans-Peter Grahsl · Voxxed Days Zurich 2019
• 46mBuilding Streaming Data Apps with Pac-ManRicardo Ferreira · All Things Open 2020
• 58mRealtime Analytics Geofencing KafkaGuido Schmutz · Voxxed Days Banff 2019
• 49mLet’s Build a Streaming Data Pipeline!Robin Moffatt · DataXDay 2020