Hail Hydrate! From Stream to Lake
summary
Claims a runnable FLaNK pipeline that uses Apache NiFi, Pulsar, Flink, MiNiFi to ingest IoT and NOAA weather feeds (NOAA ZIPs every ~15 minutes), convert XML→JSON, infer schemas, and write ORC/Parquet to S3 while also publishing to Pulsar/Kafka. Demo shows hundreds of records per node, over 25,000 queued messages, Flink SQL continuous queries exposed as materialized REST views, and on‑stream deep learning (ResNet50) via MXNet/DL4J producing bounding boxes — all running live and producing persistent lake files.
AI-written summary · about
chapters (23)
- 0:00Streaming and data lakes intro
- 2:20Agenda and use case
- 3:42High-volume streaming challenges
- 6:10Pipeline development pain points
- 6:53NiFi for scalable streaming
- 9:21FLIP stack for streaming
- 10:44Pulsar Flink connector
- 11:26NiFi features
- 13:12Pulsar for messaging
- 14:35Flink for analytics
- 15:57Integrating machine learning into streams
- 16:59Architecture overview
- 18:22NiFi live demo
- 20:50IoT real-time processing
- 23:16Weather data ingestion
- 25:44Stream to lake with ORC/Parquet
- 27:10Deep learning in NiFi
- 28:58Flink SQL analytics
- 30:45Materialized views and REST
- 32:32Monitoring Flink apps
- 33:13Data lake storage and viz
- 34:16Real-time alerts and Slack
- 34:58Q&A
related talks
47mBuilding Modern Data Streaming Apps with Open Source
39mCodeless Pipelines With Pulsar and Flink
29mHello Hydrate! From Stream to ClickHouse with Apache Pulsar and Friends
28mStreaming Lakehouse with Kafka, Flink and Iceberg
46mSmart Transit: Real-Time Transit Information with FLaNK
26mArchitecting real-time transactional data lake
34mBeaming Flink to the Cloud @ Netflix
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink