~/talk

Breaking Down Data Silos: Enterprise Data Automation with Python on Azure

Talkat PyCon India 2025 · 2025-09-12..09-15· [cloud-infrastructure, data-engineering]
slides:none on file

Built a central Azure Data Lake repository that removes direct source access and integrates 20+ systems so Python jobs (pandas, Polars, DuckDB) handle ingestion, cleaning and medallion layering (bronze/silver/gold), moving CSVs to Parquet and enforcing date/currency/ID standards. The pipeline powers Power BI reports, serverless Logic Apps for automated emails and approvals, Azure Data Factory orchestration replacing Airflow, and already runs over 70 TB of history and automates 500+ reports with alerting on failed quality checks.

AI-written summary · about

  1. 0:00Introduction and speaker intro
  2. 0:22The file v3 problem
  3. 1:25Enterprise data chaos explained
  4. 2:33Watermelon reporting and broken systems
  5. 3:39Silo vicious circle root cause
  6. 4:20Central nervous system on Azure
  7. 5:23Python in enterprise data flow
  8. 6:05Enterprise systems and data sources
  9. 6:46Medallion architecture explained
  10. 7:30Bronze raw data layer
  11. 8:57Event-driven data ingestion
  12. 9:42Timestamped data organization
  13. 10:24Batch vs delta data capture
  14. 11:55Data quality and standardization
  15. 12:37Centralized cleaning patterns
  16. 13:41Currency and terminology issues
  17. 14:23Legal vs day-to-day data terms
  18. 15:06Data quality scores and governance
  19. 15:55Silver clean data layer
  20. 16:37Enterprise semantic layer
  21. 17:17Dimensions and facts explained
  22. 18:00Data models and semantic layer
  23. 18:22Automating business processes
  24. 19:26Travel request automation example
  25. 20:28Data-driven PDFs and emails
  26. 21:13Serverless Logic Apps automation
  27. 21:58Enterprise data challenges
  28. 22:46Data catalog and glossary
  29. 23:06Data quality enforcement
  30. 23:47Performance optimizations
  31. 24:28Business process changes
  32. 25:11Excel databases reality
  33. 25:52Start small automate redundant tasks
  34. 26:36Enterprise data is wide not big
  35. 27:17Q&A introduction
  36. 27:40Fabric vs Python jobs on ADLS
  37. 29:10Data Factory vs Airflow
  38. 29:57Automating reports and emails
  39. 31:01Handling outliers and data errors
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.