Workshop 3: Data Engineering - A Roadmap... by Ramon Perez [Part 3]
summary
Walks through a working Python data pipeline that extracts CSV/parquet, cleans and one‑hot encodes columns, writes parquet and loads a table into a local analytics DB, and wires everything into reproducible stages with DVC. Uses pandas, Typer CLI, parquet files, SQLAlchemy/Ibis and DVC; combines London (~17k) and Seoul (~10k) into ~26k rows; shows Ibis to create analytics.db and DVC pipeline reproduces extract transform load while demonstrating CLI flags and error handling.
AI-written summary · about
chapters (15)
- 0:00Data analysis London Seoul
- 1:46Selecting columns for analysis
- 4:22Renaming merging data frames
- 7:16Ordering merging by timestamp
- 10:49Command-line with Typer
- 18:30Data pipeline challenges
- 20:15Loading into Ibis database
- 24:56Saving parquet or tables
- 26:00Data Version Control intro
- 28:49Tracking files DVC pipelines
- 31:00Data engineering workflow setup
- 33:27Reproducing pipelines
- 35:05Tracking data lineage updates
- 36:52Scheduling pipelines with Cron
- 37:34Closing remarks Q&A
related talks
03h 07mData of Unusual Size: Interactive Visualization
35mData Processing Pipelines for Small Big Data
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
32mSimple Data Engineering in python 3.5+ with Bonobo
01h 25mA practical guide to analysis and interactive visualization of massive datasets
29mBuilding Large Scale ETL Pipelines with Dask
53mTips and tricks for robust big data applications
44mThe Art of Data Engineering