Using the Mm FLaNK Stack for Edge AI (Flink, NiFi, Kafka, Kudu)
summary
A working edge pipeline that reads desk sensors with MiNiFi -> NiFi -> Kafka, validates and writes Avro with a central schema registry, and runs a continuous Flink SQL job that joins two Kafka topics into a global sensor events topic. NiFi also pushes joined results into Apache Kudu as a real-time data mart; Kafka Connect streams data to HDFS. Demo shows live records, schema-backed provenance, a Flink job running ~1 hour with matching counts, and end-to-end reads and queries that succeed.
AI-written summary · about
chapters (30)
- 0:00Introduction to Flank Stack
- 1:09Cloudera’s Flink Support
- 2:33Reference Architecture Overview
- 3:39Streaming Data Lifecycle
- 5:03Schema Registry Benefits
- 6:06What is the Flank Stack
- 7:10Data in Motion Platform
- 8:14CDP Data Hubs Demo
- 9:43Why Use NiFi
- 10:47Kafka in the Stack
- 11:49Flink SQL and Dashboard
- 12:10Live Demo Setup
- 12:51NiFi Data Ingestion
- 13:33Data Routing in NiFi
- 15:18Kafka Topic Inspection
- 16:42Kafka Connect Apps
- 17:27Kudu Real-time Data Mart
- 18:31Pushing Data to Kudu
- 19:13Flink Application Demo
- 21:00Combining Components
- 21:42Flink SQL Client Setup
- 23:27NiFi JSON Processing
- 24:31Schema Validation in NiFi
- 25:33End-to-End Data Flow
- 26:39Security and Governance
- 27:22Schema Versioning
- 28:27Flink SQL Table Discovery
- 29:08Flank Stack Recap
- 29:51Real-time Analytics
- 30:34Q&A and Resources
related talks
37mBuild a Flink AI Ecosystem
29mSimplifying data flows
12mUsing AI-enabled Edge Cloud Technologies to Strengthen Europe’s...
59mOptimizing Complex Workflows with Event - Driven - Multi Agentic Approach
19mDatabricks
37mBuild a Modern Data Platform on AWS
51mOptimizing Complex Workflows with Event-Driven Multi-Agentic Approach
20mAI