Apache NiFi 101: Introduction and Best Practices
summary
Introduction to Apache NiFi shows a runnable pipeline: start on laptop or Docker, run on Kubernetes with external storage, and scale from a single node to 100 clients and "billions of events a second" without code changes. Live demo ingests files from FTP (Jetson), runs a custom Java processor that calls MXNet for image ML, writes results to S3 and posts to Slack — the demo works and NiFi doesn't crash in production. Covers flow file model, durable provenance, queue backpressure, Record/Calcite readers, Avro/Parquet and controller services.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 2:20Requirements
- 5:45Flow File Metadata
- 7:32Processors
- 9:24Connections
- 12:08Reader API
- 13:02Cache
- 14:14Provenance
- 15:20Command Line Toolkit
- 15:52Python Wrapper
- 16:32Consuming from Things
- 17:08Common Use Cases
- 17:58Navigation
- 19:00Parameters
- 20:26Controllers
- 22:44Navigating
- 26:56Flow File
- 27:23Metadata
- 28:24Counters
- 30:09Statement Functions
- 30:49Custom Processors
- 32:39Programming
- 35:29Validate
- 37:19Reporting
- 38:24Adding Sources
- 39:57Errors
related talks
21mMeeting complex data load and data preparation challenges
47mBuilding Modern Data Streaming Apps with Open Source
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data
42mConnecting the data infrastructure with the DataFlow
39mApache Iceberg ingestion with Apache NiFi
30mThe Fabulous World of Data – ETL Powered by Apache NiFi and Scrapy
29mBuilding Apache NiFi 2.0 Python Processors
59mPolyglot Pipelines with Apache Nifi