Data Migration using Azure Data Factory
summary
Migration proof-of-concept moves thousands of Azure SQL tables into an Azure Data Lake Gen2 storage account by converting each table into per-table CSVs, using an Azure Data Factory pipeline with Lookup → ForEach → Copy pattern, linked services to Azure SQL and ADLS Gen2, and a sample AdventureWorks database for demo. The implemented flow supports scheduled fortnightly incremental runs for gradual cutover, scaled to a multi-vendor $30M migration and the pipeline executes successfully in the demo. per-table CSV files scheduled triggers
AI-written summary · about
chapters (7)
- 0:00Professional introduction
- 1:11Use case and business problem
- 3:36Data architecture overview
- 7:52Infrastructure setup
- 14:44Configuring data movement
- 24:19Pipeline construction
- 29:38Execution and validation
related talks
41mA Case Study - Data Movement and Transformation using Azure Data Factory
50mAzure Data Factory patterns and best practices
01h 00mBuilding cloud-native data pipelines using Azure Data Factory
49mYour first flight with Data Factory in Microsoft Fabric
50mDeep {sky}diving into Data Factory in Microsoft Fabric
18mMigrating my favorite past project from On Premise to Azure
50mMigrating data solutions to the cloud - a checklist.
56mMigrate Databases to the Cloud - Why, How and Best Practices