Use Apache-Airflow to Build Data Workflows

on YouTube ↗38m

" summary

Claims a reproducible, versioned pipeline built with python and Apache-Airflow that replaces brittle cron jobs and tracks state in a metadata database, scheduler and web UI. Shows a working connector to the Twitter API, stores tweets in a local MySQL, and pushes outputs to CSV or S3. Demonstrates failures and scale — cron job ran for weeks, 6 GB of tweets filling storage, and Twitter now returns only 30 days of Twitter data so airflow scheduling and DAG logic actually matter.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 2:10Data science challenges
3. 6:25Twitter sentiment problem
4. 8:09Airflow for pipelines
5. 13:05Airflow concepts
6. 16:20DevOps with Airflow
7. 20:58Airflow web server demo
8. 27:05Twitter data analysis
9. 32:48Q&A

" related talks

• 36mGetting Started with Apache Airflow - Building Your First WorkflowRicardo Sueiras · Big Data Conference Europe 2023
• 36mElegant data pipelining with Apache AirflowBolke de Bruin · PyData Amsterdam 2018
• 42mHow I learned to time travel, or, data pipelining and scheduling with AirflowLaura Lorenz · PyData DC 2016
• 32mOrchestrating Your Data Pipelines with Apache AirflowRicardo Sueiras · GOTO Amsterdam 2022
• 34mData Ingestion in Python using Apache AirflowSowmya Ganapathi Krishnan, Yew Wee · PyCon APAC 2018
• 39mSell cron, buy Airflow: Modern data pipelines in financeJames Meickle · PyData DC 2018
• 01h 25mApache Airflow in the Cloud: Programmatically orchestrating workloads w/ PySatyasheel, Kaxil Naik · PyData London 2018
• 48mFlow Is in the Air: Best Practices with Apache AirflowDominik Benz · PyCon.DE 2017