Building Modern Data Streaming Apps with Open Source
summary
Shows a working pipeline that ingests REST APIs, transit feeds and ADS‑B aircraft data with Apache NiFi, publishes to Kafka or Apache Pulsar, runs continuous queries with Flink SQL, and writes results to Iceberg and Pinot. Live code demonstrates transforming JSON/Avro, splitting streams, filtering and enriching, and sending thousands of records per second from laptop prototypes to Kubernetes. Includes schema registry use, materialized REST views for non‑Kafka clients, and concrete DevOps tips for backups, provenance and deployment.
AI-written summary · about
chapters (4)
- 0:00Start
- 14:21Messages - the Basic Unit of Apache Pulsar
- 16:26Apache Pulsar Ecosystem
- 25:47FDLC: Flow Development Lifecycle
related talks
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data
39mCodeless Pipelines With Pulsar and Flink
36mHail Hydrate! From Stream to Lake
46mApache NiFi 101: Introduction and Best Practices
16mLife is But a (Data) Stream: Building Quality Data Pipelines
14mFast Data on-Ramp with Apache Pulsar on K8
43mBuild a Modern, End-to-End, Big Data Scala Reference Application
15mFrom Air Quality to Aircraft and Automobiles, Data Is Everywhere