Practical Pipelines: A Houseplant Alerting System with ksqlDB.
summary
Build uses a Raspberry Pi and cheap I2C capacitive moisture sensors feeding Avro-encoded readings into a managed Apache Kafka cluster, with ksqlDB doing stateful joins and windowed aggregates and Kafka Connect pushing Telegram messages. Sampling runs every 30 seconds for ~40–50 plants, but sensors’ I2C sensors limited to four addresses forces a mapping workaround; ksqlDB counts low readings over non-overlapping six-hour windows and requires 120 low samples before alerting, using emit final windowing to avoid repeated pings — alerts arrive via Telegram and the pipeline works end-to-end.
AI-written summary · about
chapters (15)
- 0:00Introduction and motivation
- 3:54What is Apache Kafka
- 5:40Hardware setup and sensors
- 7:45Defining data schemas
- 9:51Producing data to Kafka
- 11:57Storing metadata in Kafka
- 13:01Using Telegram for data entry
- 16:09Dynamic sensor mapping
- 17:56Consuming sensor mappings
- 20:25Stream processing options
- 21:48Setting up ksqlDB tables
- 23:13Joining data streams
- 25:19Alert logic with windows
- 28:32Sending alerts via Kafka Connect
- 29:55Key takeaways and resources
related talks
52mPractical Pipelines: A Houseplant Alerting System with ksqlDB
53mPractical Pipelines: Houseplant Soil Alerting System with ksqlDB
01h 05mPractical Pipelines: A Houseplant Soil Alerting System with ksqlDB
45mPractical Cloud Pipelines
20mSet your data streams in motion in the ocean!
47mBuilding a Telegram bot with Apache Kafka and ksqlDB
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
31mEnd-to-end Flink SQL with Schema Registry, Kafka & Kudu