Evolution of Extract-Transform-Load (ETL) processes towards data product pipelines
summary
Compares two implemented pipelines: a discovery pipeline that extracts heterogeneous metadata, maps it to a common schema and saves it in Apache Atlas, and an ingestion pipeline that profiles sampled data, computes quality KPIs and updates Atlas. Uses a Python-native orchestration tool (Mate/Mage) to run jobs and evaluates OpenMetadata vs Atlas for data-product semantics. Integrates these flows into DATAMITE and produces Gaia‑X‑compliant JSON descriptors, showing publishing via data‑space connectors works but metadata harmonization remains the main obstacle.
AI-written summary · about
chapters (8)
- 0:00ETL evolution and data products
- 1:54Data as monetizable asset
- 4:23Unlocking data value challenges
- 6:12Traditional ETL to modern pipelines
- 9:05DataMight architecture overview
- 11:59Data ingestion and metadata enrichment
- 13:47Monetizing data products with Gaia X
- 15:37Q&A
related talks
24mMetadata driven framework for data discovery and data quality
31mOne Data Pipeline to Rule Them All
16mDemocratizing Data Pipelines: Introducing the AI-Powered ETL Assistant
34mTales of Data Architecture Evolution
16mData pipeline automation with built-in metadata and lineage (PRT054)
41mModern Data Architectures – What They Are and How to Implement?
34mData pipelines powered by Open source and fun
39mData Pipeline Evolution