Building Real-time Travel Alerts
summary
Builds a real pipeline that ingests global GTFS‑RT and RSS, converts protobuf and CSV to JSON, pushes into Apache NiFi, then to Apache Kafka and streaming SQL in Apache Flink to produce a REST materialized view and dashboards. Alerts go to Slack/Discord, maps render with leafletjs, and examples handle surprising volumes — hundreds of thousands of records (250k paused, ~500k processed) and spikes of 20,000/sec on a laptop — proving the stack works end to end.
AI-written summary · about
chapters (14)
- 0:00Real-time travel alerts intro
- 4:02Why real-time transit data matters
- 9:40Real-time app architecture
- 13:54Introducing Flank stack
- 21:09Kafka basics
- 22:14Flink for streaming analytics
- 24:23NiFi features and benefits
- 27:46NiFi live demo
- 31:22RSS to alerts processing
- 35:17GTFS-RT data ingestion
- 42:11Authenticated transit systems
- 43:17Brazil transit alerts demo
- 47:43Halifax transit enrichment
- 51:59Dashboards from streaming data
related talks
47mBuilding High Performance Real-time Stream Processing Microservices
45mWhy Spring Belongs In Your Data Stream (From Edge to Multi Cloud)
28mPowering Crowd-sourcing of Public Transit Delays with Rust
57mUsing Kafka for Real-Time Data Ingestion with .NET
31mStreaming: Real-Time Architecture for Connected Products in Automotive
06mPREVIEW: Real-time Stream Processing at Scale
28mManaging and Scaling Real Time Data Pipelines using Go
28mTransforming the Financial Services Industry with Event Stream Data