Data Warehousing with Python
summary
Describes a Python-based data warehouse built from open-source components: PostgreSQL as the primary store, pip-packaged Python pipelines, and a lightweight execution layer that splits computation into parallel jobs. Explains how ETL tasks are sharded, indexed and scheduled to handle real-world sizes (e.g., a 25 GB table and multi-hundred-GB datasets), streams input without loss on commodity servers, and converts multi-day batch runs into shorter incremental executions for marketing dashboards and automation.
AI-written summary · about
chapters (9)
- 0:00Data warehousing with Python
- 3:03Data warehouse complexity
- 9:56Demo and pipeline overview
- 13:02Customer journey processing
- 18:12PostgreSQL and ETL tools
- 22:31Local transformation efficiency
- 27:37Pipeline conventions and public data
- 31:03Security in data systems
- 34:13Summary
related talks
32mHow to Build a Data Warehouse?
43mData warehousing reinvented for today's needs
39mHow Python Drives Modern Data Engineering
40mPyData Stack: Pure Python open source data platforms
27mBuilding a PostgreSQL data warehouse
32mScalable Python and SQL Data Engineering without Migraines
33mData Engineering simplified with Snowpark for Python on Snowflake
42mWeb Development To Big Data. A Journey