What will remain of my website in 50 years?
summary
French legal deposit for the web requires the BnF to collect websites based on domain, producer location or content origin using the open-source crawler Heritrix3 with per-site budgets (number of URLs), depth, frequency and filters. Captures are stored as WARC files with deduplication; in 2023 the service crawled about 5.8 million domains (~6.3 billion URLs that year), totaling ~55 billion URLs and ~2 PB since 1996, and access is provided only at the BnF or partner libraries.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:44BNF missions
- 2:34Legal web deposit
- 7:09Heritrix 3 robot
- 10:00How collection works
- 12:50Accessing archives
- 13:54Demo and conclusion
related talks
28mBiodiversity and Web Futures
22mOpen Web Search: A platform for a free European Web Index
49mInternet Archive: More than the Wayback Machine
35mModern Web Data Extraction: Techniques, Tools, Legal and Ethical Considerations
14mTHE END OF THE PAPER INTERNET
11mHow to stop worrying & collect early banner ads!
14mEverything Always Everywhere
43mFrontera: open source, large scale web crawling framework