AI-Powered ETL Pipeline Orchestration: Multi-Agent Systems in the Era of Generative AI
summary
Claims a working prototype that ties Apache Airflow, Kubernetes and an LLM-driven stack into agentic DAGs that run extraction, validation, AI-driven cleaning and loading with a supervisor/router and retry agents; demo uses a GlobalShop-style e-commerce flow and shows generative models filling missing values, matching schema changes, and predictive scheduling to adjust resources. Implementation options include LangChain/LangGraph for multi-agent orchestration and TensorFlow for models; scaling, latency, integration with legacy systems and data governance remain the main hurdles.
AI-written summary · about
chapters (14)
- 0:00Introduction to ETL
- 1:00ETL concepts and history
- 4:48Limitations of Cron jobs
- 6:17Apache Airflow for ETL
- 9:11AI in data pipelines
- 11:12Multi-agent systems
- 13:27Agentic DAGs in Airflow
- 14:32Introduction to LangGraph
- 17:30LangGraph ETL example
- 19:00LangGraph for orchestration
- 20:03Evolution of ETL tools
- 21:25Challenges of AI in ETL
- 22:56Future of ETL
- 24:30Q&A session
related talks
17mAI TechOps: Data and Infra for AI Foundation to Scale and Manage 1000s of Cohesive Agents
27mFrom Chatbots to Agents: Orchestrating Agentic AI on Existing Enterprise Kubernetes Infrastructure.
01h 29mBuild Your Own Data Science AI Agents
29mLeveraging AI to Enhance DevEx & Operational Efficiency
50mHarnessing Event-Driven and Multi-Agent Architectures for Complex Workflows in Gen...
34mOrchestrating Generative AI Workflows to Deliver Business Value
29mAPI Integration augmented by GenAI
45mHow to Really Scale AI: Combining Traditional and Agentic Automation