Data Discovery with Amundsen
summary
Amundsen is an open source data discovery system that indexes metadata into a Neo4j graph backend and an Elasticsearch search layer, ingests via automated Airflow extractors (Postgres Hive BigQuery etc), surfaces owners, lineage and column previews, and supports bookmarking and notifications. The claim is a 30% data scientist productivity gain from faster discovery; deployment requires extractor work and UI customization but the system demonstrably links tables dashboards and people and enables downstream impact checks.
AI-written summary · about
chapters (14)
- 0:00About the team
- 2:24Data discovery Issues
- 5:43Side effects
- 6:41Metadata is key to next bigdata w
- 8:19ABC of metadata
- 9:54What do we need?
- 11:32What's out there?
- 12:21Important things for the journey
- 19:11Relations rather than entities
- 20:17SQL vs Neo4j queries
- 22:11ElasticSearch for search and relev
- 22:42Metadata extractors
- 24:00What to expect...
- 26:38Roadmap
related talks
24mMetadata driven framework for data discovery and data quality
40mData Intelligence Platform on AWS
27mRecipe for Building a Discoverable and Governed Data Platform.
17mDemocratising data analytics
15mGraphRAG-Powered AI Agents: Transforming Corporate Metadata into Intelligent Knowledge
55mDemocratize data: The intersection of governance and analytics
15mEasy Metadata Management (Git for Data) Using Neo4j
28mSearching for Research Fraud in OpenAlex with Graph Data Science