Apache NiFi, Kafka, RisingWave, Iceberg + Stock Data
summary
Live pipeline demonstrates real-time stock ingestion from websockets and APIs into Apache NiFi and Python processors, publishes JSON to Kafka, materializes batches into Parquet via Apache Iceberg (writes every 1,000 rows) and runs incremental SQL queries with RisingWave for real-time analytics and materialized view updates. Code runs on a laptop with Docker, MinIO and SQLite catalog, supports Slack-driven lookups to convert company names to tickers, and the demo shows the pipeline actually running and answering live stock queries.
AI-written summary · about
chapters (12)
- 0:00intro
- 0:20preamble
- 0:54what is stream processing?
- 3:47understanding stream processes
- 6:15what is risingwave
- 8:40data stream processes in risingwave
- 10:02streaming etl
- 10:27real-time analytics
- 12:55event-driven applications
- 13:21the architecture
- 14:12using apache nifi, apache kafka, risingwave, and apache iceberg with stock data and llm
- 27:24thank you
related talks
47mBuilding Modern Data Streaming Apps with Open Source
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
46mApache NiFi 101: Introduction and Best Practices
39mApache Iceberg ingestion with Apache NiFi
15mFrom Air Quality to Aircraft and Automobiles, Data Is Everywhere
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
59mPolyglot Pipelines with Apache Nifi
39mCodeless Pipelines With Pulsar and Flink