IoT and Stream Processing
summary
Edge stream processor built on fluent-bit runs on embedded Linux in C with a ~500 kilobytes footprint and exposes SQL-like queries, windows and simple forecasting. It processes 1 million records in demos, computes per-country averages every 5s and forecasting every 10s, and sends only aggregated results (two outgoing messages instead of forwarding 1M), cutting bandwidth and cloud ingestion. Plugins support inputs, filters (Lua), outputs (Go/C), forward protocol and retries; implementation is production-used and demonstrates low-latency edge analytics with successful demos.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:27Challenges
- 6:01Stream Processing
- 8:26How Stream Processing Works
- 11:19Logging
- 16:58Performance
- 18:49Implementation
- 20:19First Demo
- 23:37Second Demo
- 34:07Questions
- 40:43Plugins
- 42:29Internal IO
related talks
43mCreating an IoT Data Layer for Collecting, Storing, Analyzing
43mHail Hydrate! From Stream to Lake Using Open Source
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
31mStreaming Graphs, Because We Can't Afford to Query Any More
45mDeep dive into Unbounded Data Processing Systems
07mTransforming Big Data into Real-Time Product Insights
28mFluentd: Data Streams in Ruby World
16mEdge to Cloud with InfluxDB