Managing a search engine for over 600 million openly licensed media records
summary
Openverse is a search engine that aggregates — not hosts — openly licensed images and audio, claiming over 600 million records and integrated into WordPress. The stack uses Elasticsearch + Postgres + Airflow with a bespoke Falcon refresh server; ingestion normalizes >30 provider APIs, copies tables and swaps indices in a weekly table-and-index swap to push new data. It delivers attribution text, but the popularity-normalization step becomes a multi-day bottleneck as data grows, so plans target decoupling popularity, moving to ELT, and exploring vectorized relevance.
AI-written summary · about
chapters (8)
- 0:00Openverse introduction
- 2:26Openverse licensing
- 7:44Role of the commons
- 13:24Technical stack overview
- 16:10Data ingestion challenges
- 23:55Data architecture reimagined
- 32:20Community contributions
- 36:34Q&A
related talks
20mMeasuring Data with OpenSearch to Better Serve the Community
22mOpen Web Search: A platform for a free European Web Index
13mSecuring the Data Frontier: Real-World Strategies With OpenSearch
46mDelivering the next wave of innovation with OpenSearch
38mMigrating from Self-Managed ElasticSearch to OpenSearch
06mSponsored Keynote: Foundation-Led Innovation: OpenSearch's Impact on Modern Data I...
35mPlatformizing OpenSearch at Uber
55mVector Search Made Simple: Getting Started with OpenSearch for AI Applications