Breaking Down Data Silos: Enterprise Data Automation with Python on Azure
summary
Built a central Azure Data Lake repository that removes direct source access and integrates 20+ systems so Python jobs (pandas, Polars, DuckDB) handle ingestion, cleaning and medallion layering (bronze/silver/gold), moving CSVs to Parquet and enforcing date/currency/ID standards. The pipeline powers Power BI reports, serverless Logic Apps for automated emails and approvals, Azure Data Factory orchestration replacing Airflow, and already runs over 70 TB of history and automates 500+ reports with alerting on failed quality checks.
AI-written summary · about
chapters (39)
- 0:00Introduction and speaker intro
- 0:22The file v3 problem
- 1:25Enterprise data chaos explained
- 2:33Watermelon reporting and broken systems
- 3:39Silo vicious circle root cause
- 4:20Central nervous system on Azure
- 5:23Python in enterprise data flow
- 6:05Enterprise systems and data sources
- 6:46Medallion architecture explained
- 7:30Bronze raw data layer
- 8:57Event-driven data ingestion
- 9:42Timestamped data organization
- 10:24Batch vs delta data capture
- 11:55Data quality and standardization
- 12:37Centralized cleaning patterns
- 13:41Currency and terminology issues
- 14:23Legal vs day-to-day data terms
- 15:06Data quality scores and governance
- 15:55Silver clean data layer
- 16:37Enterprise semantic layer
- 17:17Dimensions and facts explained
- 18:00Data models and semantic layer
- 18:22Automating business processes
- 19:26Travel request automation example
- 20:28Data-driven PDFs and emails
- 21:13Serverless Logic Apps automation
- 21:58Enterprise data challenges
- 22:46Data catalog and glossary
- 23:06Data quality enforcement
- 23:47Performance optimizations
- 24:28Business process changes
- 25:11Excel databases reality
- 25:52Start small automate redundant tasks
- 26:36Enterprise data is wide not big
- 27:17Q&A introduction
- 27:40Fabric vs Python jobs on ADLS
- 29:10Data Factory vs Airflow
- 29:57Automating reports and emails
- 31:01Handling outliers and data errors
