Orchestrating success
summary
Vinted coordinates over 20 decentralized data teams handling petabytes by building a Python generator on top of Apache Airflow that renders DAGs from a simple scheduleuler YAML, emits DAG files to GCS, and enforces contracts via a centralized asset registry. DBT models, Vertex AI jobs and dockerized tasks become validated in CI, SLOs map to Airflow sensor timeouts, external task sensors manage cross-team dependencies, and generator versioning lets the platform upgrade by rolling forward rather than changing every repo.
AI-written summary · about
chapters (8)
- 0:00Orchestrating success at Vinted
- 3:33Monolithic data pipeline challenges
- 7:05Decentralizing teams with coordination
- 8:32Generator in practice
- 16:18Developer workflow and CI/CD
- 21:57Automated alerts and downstream tasks
- 24:01Centralized abstraction benefits
- 25:26Q&A
related talks
30mToo Big for DAG Factories?
28mBuilding Self-Service Data Platforms
22mData pipelines != workflows: orchestrating data with Dagster
36mElegant data pipelining with Apache Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
38mUse Apache-Airflow to Build Data Workflows
30mUsing Gremlin API and Apache Airflow to facilitate Complex System Integrations
41mModular & Reusable: A Recipe to Scaling Data Across a Portfolio