Practical Pipelines: A Houseplant Alerting System with ksqlDB
summary
A home system uses a Raspberry Pi and I²C capacitive soil sensors that can only be addressed four ways, producing moisture and temperature readings every 30 seconds into Apache Kafka; plant metadata and readings use Avro schemas and Schema Registry. ksqlDB joins the stream and a metadata table, applies a six‑hour tumbling window and counts >=120 low readings using emit final to produce at most one alert per plant per window. Alerts flow out via Kafka Connect HTTP sink to Telegram. I²C sensors limited to 4 addresses, 30s readings, 6‑hour window, 120 low counts, Avro schemas + Schema Registry.
AI-written summary · about
chapters (22)
- 0:00Start
- 5:34Paradigm Shift
- 6:53Thinking in Events
- 9:41Kafka Storage
- 13:06Kafka Cluster
- 14:39Kafka Replication
- 15:48A Practical Pipeline Review
- 17:04The Hardware
- 18:35The Software
- 21:44Writing Data to Kafka
- 24:05Processing with Kafka
- 27:18Creating a Table
- 28:52Creating a Stream
- 29:18Stream Enrichment
- 33:15Windowing and Computing Alerts
- 36:18Alerting with Telegram
- 39:09Conversation Handler
- 40:16Producer Code
- 41:43Updating Sensor Mappings
- 42:13Consuming Mapping Updates
- 44:33A More Practical Pipeline
- 45:16Streaming Data Pipeline
related talks
31mPractical Pipelines: A Houseplant Alerting System with ksqlDB.
53mPractical Pipelines: Houseplant Soil Alerting System with ksqlDB
35mPractical Pipelines: A Houseplant Alerting System with ksqlDB
01h 05mPractical Pipelines: A Houseplant Soil Alerting System with ksqlDB
45mPractical Cloud Pipelines
20mSet your data streams in motion in the ocean!
31mFun with Sensors
56mTaming IoT Data: Making Sense of Sensors with SQL Streaming