Taming IoT Data: Making Sense of Sensors with SQL Streaming
summary
End-to-end IoT pipeline built with Apache Kafka and KSQL parses replayed July 2018 smart‑home sensor and weather streams (two households, two weather stations) replayed faster than real time, runs windowed aggregations (1h tumbling, 10min for dashboards) that produce concrete results like 38 events and 0.6 kWh in one hour, persists outputs to MongoDB via Kafka Connect using five connectors, exposes live Server‑Sent Events with a Spring WebFlux app and feeds Power BI dashboards, all working in the demo with KSQL (Kafka SQL streaming engine).
AI-written summary · about
chapters (21)
- 0:00Start
- 2:05What Is the Stream Processing
- 9:28Stream Processing
- 12:38User-Defined Functions
- 14:31Interactive Mode
- 15:16Interaction Main Mode
- 17:20The Headless Mode
- 21:04Consuming Applications
- 21:48Ingesting some Sensor Data
- 24:23Define a Table
- 25:56Limit Clause
- 26:28Filters
- 27:17Derived Streams
- 30:39Processing Options
- 32:30Aggregations Based on Time Windows
- 44:40Hourly Consumption
- 45:34Live Dashboards
- 45:52Streaming Dashboards of Power Bi
- 46:50Power Consumption
- 52:08Session Based Windows
- 55:26How Did the Data Find Its Way into the Corresponding Topic
related talks
31mFun with Sensors
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
49mLet’s Build a Streaming Data Pipeline!
08mPREVIEW: Kafka Stream Processing for Everyone with KSQL
58mRealtime Analytics Geofencing Kafka
01h 01mStream Data Processing for Fun and Profit
39mModern Event-driven Architecture: Adopting Data in Motion