Polyglot Pipelines with Apache Nifi

on YouTube ↗59m

" summary

Shows a working pipeline that moves CSVs from the Chicago data portal into OpenSearch using Apache NiFi and Apache Tika, then runs a Python-based table detection model (Hugging Face Transformers + PyTorch + OpenCV) via py4j style IPC inside a per-plugin virtualenv. Demonstrates annotated images and COCO JSON output, packaged as a deployable Nar file (built with hatch), notes GPU support via Python libs and serialization/packaging tradeoffs.

AI-written summary · about

" chapters (7)

1. 0:00Introduction and speaker background
2. 2:49Apache NiFi crash course demo
3. 14:04Why mix Java and Python
4. 21:08Py4J integration in NiFi
5. 29:56AI use case with Python processors
6. 41:51Building and deploying Python processors
7. 48:20Q&A on performance and GPU support

" related talks

• 41mAdding Generative AI to Real Time Streaming PipelinesTimothy J Spann · Build Stuff 2024
• 28mApache NiFi, Kafka, RisingWave, Iceberg + Stock DataTimothy J Spann, Karin Wolok · Conf42 Python 2024
• 29mBuilding Apache NiFi 2.0 Python ProcessorsTimothy J Spann · Conf42 Python 2024
• 46mApache NiFi 101: Introduction and Best PracticesTimothy J Spann · ApacheCon@Home 2021
• 30mThe Fabulous World of Data – ETL Powered by Apache NiFi and ScrapyAdrian Hishow · code.talks 2022
• 34mData pipelines powered by Open source and funSayantika Banik · stackconf 2022
• 47mBuilding Modern Data Streaming Apps with Open SourceTimothy J Spann · Open AI + Data Forum @ Open Source Summit North America 2023
• 21mMeeting complex data load and data preparation challengesJohannes Peter · Berlin Buzzwords 2018