Flink for Everyone: Self-Service Data Analytics with StreamPipes

on YouTube ↗33m

" summary

StreamPipes is an open-source self‑service pipeline builder that runs processors as Docker containers and deploys jobs to Apache Flink with Kafka as the broker. The UI creates MQTT→Kafka adapters, an aggregation (5s window, output each 1s) and a trend detector that triggers a push notification and writes statistics into IoTDB. Developers add processors via an SDK and wrappers (Java, Flink, Kafka Streams, Python in progress). Demonstrations deploy to a Flink cluster, detect a 200% drop in 5s, and persist results.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to StreamPipes
2. 2:17Need for self-service IoT analytics
3. 7:02Technical architecture overview
4. 10:13Demo connecting and visualizing data
5. 16:53Extending algorithm toolbox with SDK
6. 24:31Demo detecting leaks and storing stats
7. 30:22Lessons learned and getting started

" related talks

• 38m7 reasons to use Apache FlinkJakub Piasecki, Fabian Hueske · Berlin Buzzwords 2019
• 56mTaming IoT Data: Making Sense of Sensors with SQL StreamingHans-Peter Grahsl · Voxxed Days Cluj-Napoca 2019
• 44mAn Introduction to Edge Extensions, Client API and Data ExplorerPatrick Wiener, Dominik Riemer · ApacheCon@Home 2021
• 31mFun with SensorsYogi Rampuria · Distributed SQL Summit Asia 2022
• 20mSet your data streams in motion in the ocean!Warren Vella · DataEngBytes 2022
• 34mBuilding and Scaling Robust Zero-code IoT Streaming Data PipelinesPaul Brebner · ApacheCon@Home 2021
• 20mBuilding realtime efficient queries in an IOT platformDinesh Narayanan · Data By the Bay 2016
• 35mPractical Pipelines: A Houseplant Alerting System with ksqlDBDanica Fine · PyCon Italia 2023