Using Gremlin API and Apache Airflow to facilitate Complex System Integrations
summary
Builds a graph-based integration layer using Azure Cosmos DB Gremlin API and orchestrates continuous updates with Apache Airflow, containerized via Docker, deployed from GitHub to Azure Container Registry. Targets seven to eight separate work-management systems, replaces dozens of SQL join tables with explicit nodes/edges, and runs about 23 DAGs to poll, transform and push data. Claims a working model in two to three weeks, the graph functions as both a cross-system map and an operational data plane enabling analytics and external integrations.
AI-written summary · about
chapters (13)
- 0:00Intro
- 3:58Agenda
- 4:46What is a graph
- 6:10Gremlin
- 6:47Gremlin Model
- 9:16SQL Model
- 12:36Traversing the Graph
- 14:17Data Structure
- 18:36Implementation
- 19:07Apache Airflow
- 22:22Continuous Development Architecture
- 25:40Conclusion
- 26:57Questions
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
35mOrchestrating success
22mOrchestrate your data pipelines using Apache Airflow
27mAI-Powered ETL Pipeline Orchestration: Multi-Agent Systems in the Era of Generative AI
57mGraphing Your Way Through the Cosmos: Common Data Problems Solved with Graphs
34mData Ingestion in Python using Apache Airflow
52mBetter ETL with Managed Airflow in ADF
28mBuilding Self-Service Data Platforms