Build a Modern, End-to-End, Big Data Scala Reference Application
summary
A runnable Scala reference pipeline implements IoT truck sensors, joins truck and traffic streams, flags violations and computes average speed, then persists results for sub-second queries and batch ML. It wires Apache NiFi for visual flow routing into Kafka + Avro + Schema Registry, uses a streaming runtime (Storm/Spark bridges via Streaming Analytics Manager), stores aggregates in Druid + Superset for real-time dashboards, and ships a fully working repo with a single‑node 8–10GB sandbox.
AI-written summary · about
chapters (15)
- 0:00Introduction and setup
- 1:52Talk focus changes
- 3:39Modern big data pipeline challenges
- 5:06Use case and architecture overview
- 7:19Apache NiFi for data routing
- 10:20NiFi demo walkthrough
- 15:43NiFi capabilities and questions
- 19:18Schema registry for data consistency
- 25:47Streaming analytics with SAM
- 27:55Streaming Analytics Manager demo
- 31:36Data persistence with Druid
- 33:04Visualizing data with Superset
- 36:51Real-time analytics demo
- 39:31Predictive analytics overview
- 42:31Resources and conclusion
related talks
47mBuilding Modern Data Streaming Apps with Open Source
31mOne Data Pipeline to Rule Them All
42mBuilding Robust Streaming Data Pipelines with Apache Spark
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
15mFrom Air Quality to Aircraft and Automobiles, Data Is Everywhere
05mInterview
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data