Open Web Search: A platform for a free European Web Index
summary
EU-funded project builds an open web index as public infrastructure, using a distributed crawler (OWLer) based on StormCrawler and URLFrontier, storing raw crawls as WARC and C .pet files and exposing custom slices via a public dashboard and a pull tool. Infrastructure is heterogeneous across data centers, the system currently captures about 40 million pages per day, focuses on HTML text (no headless JS rendering yet), applies heuristics and public blacklists to reduce spam, and supports opt-outs and transparent crawl logs.
AI-written summary · about
chapters (6)
- 0:00Introduction to the project
- 0:59Architecture and technology
- 9:37Crawling process overview
- 11:08Data handling and formats
- 13:59Q&A and technical discussion
- 21:14Closing remarks
related talks
45mManaging a search engine for over 600 million openly licensed media records
21mURL Frontier, an open source API and implementation for crawl frontiers
48mA Journey to Open Data
40mOpen source software could save libraries
01mOpenSearchCon Europe 2026 Highlight Video
40mBuilding a Search Engine for Open Content
12mKeynote: OpenSearch, Innovation and Community
36mArkisto: an open-source, standards-based framework for digital preservation