Scaling ETL Migrations to 300x with AI
summary
Shows an agent-based pipeline that combines an input parser, Gemini Flash 2.5, a canonical JSON intermediate, and output parsers to auto-translate legacy ETL exports into target code, documentation and Terraform infra. Validation uses YAML-driven table and column tests (counts, means, nulls, duplicates) and an agent that auto-generates config, plus a synthetic data agent for end-to-end testing. Translation time drops from ~15 days to under 3 minutes and full migration from 5–7 days to under an hour.
AI-written summary · about
chapters (6)
- 0:00ETL migration challenges
- 4:54AI-assisted ETL translation
- 8:26Validation for reliability
- 17:59Automating validation
- 20:51Agentic ETL workflow
- 35:23Results and improvements
related talks
16mBeyond ETL: Revolutionizing Data Engineering Workflows
42mEnd-to-end pipeline agility
27mAI-Powered ETL Pipeline Orchestration: Multi-Agent Systems in the Era of Generative AI
24mRetail Transactions at scale for AI - From PoC to production
16mDemocratizing Data Pipelines: Introducing the AI-Powered ETL Assistant
15mNext Gen ETL Generators
16mEvolution of Extract-Transform-Load (ETL) processes towards data product pipelines
24mArgonauts of Data: Building Scalable and Effective Data Pipelin...