The Fabulous World of Data – ETL Powered by Apache NiFi and Scrapy

on YouTube ↗30m

" summary

Presents using Apache NiFi and Scrapy for ETL in data-science projects: Apache NiFi provides scalable, flow-based data processing and orchestration, while Scrapy is a fast, extensible Python web-crawling framework. Shows how each tool can collect and process data and how they can be combined to build scalable data ingestion and preprocessing pipelines.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:38Agenda
3. 2:35NiFi and Scrapy Introduced
4. 7:41Features I
5. 15:47NiFi - Essential Components
6. 16:57Building a Spider
7. 23:50Connecting NiFi and Scrapy
8. 27:46Logs, CRM, ERP and Shop Systems

" related talks

• 46mApache NiFi 101: Introduction and Best PracticesTimothy J Spann · ApacheCon@Home 2021
• 59mPolyglot Pipelines with Apache NifiBob Paulin · Devnexus 2025
• 21mMeeting complex data load and data preparation challengesJohannes Peter · Berlin Buzzwords 2018
• 47mBuilding Modern Data Streaming Apps with Open SourceTimothy J Spann · Open AI + Data Forum @ Open Source Summit North America 2023
• 28mApache NiFi, Kafka, RisingWave, Iceberg + Stock DataTimothy J Spann, Karin Wolok · Conf42 Python 2024
• 44mApache NIFi Deep Dive 300Timothy J Spann · ApacheCon@Home 2021
• 42mConnecting the data infrastructure with the DataFlowPere Urbón-Bayes · Berlin Buzzwords 2018
• 45mFrom telescope to data centre: adventures in astronomical data pipelinesSimon O'Toole · DataEngBytes 2021