Creating an ADF Pipeline Management System
summary
A pipeline orchestration system built around an Azure SQL queue stores job rows, parameters and allowed concurrency and handles tens of thousands of ADF runs per year for a 10‑person company. Two Logic Apps run (process queue every three minutes; monitor statuses every minute), an Azure Function calls the Data Factory SDK to submit pipelines and returns the run ID, and stored procedures gate concurrency and update status. The system is implemented, running, and reduces brittleness.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 2:45Company Overview
- 3:45Pipeline Issues
- 5:59Pipeline Concepts
- 8:20Pipeline Design
- 12:25Triggers
- 17:55Queue
- 21:30Implementation Details
- 27:36Active Pause
- 29:38Data Factory
- 34:23Logic Apps
- 35:41Stored Procedures
- 38:48Staggered Starts
- 40:22Data Factory Connector
- 43:27Run ID
- 45:28In Progress Label
- 46:01Get Pipeline Info
- 47:58Queue Management
- 51:52QA
- 52:52Data Factory Helper
- 53:31Contact Information
related talks
07mOptimizing Data Pipelines with Azure Data Factory
01h 05mWorkflows of Highly Functional App & Data Engineering Teams
41mA Case Study - Data Movement and Transformation using Azure Data Factory
50mAzure Data Factory patterns and best practices
01h 07mApplication Lifecycle Management in Azure Data Factory
01h 00mBuilding cloud-native data pipelines using Azure Data Factory
01h 01mServerless Azure Data handling
12mShort On Time? Don't Worry, We'll Have Your Azure Data Factory Automated in Only 20 Minutes!