Data & Metadata at the Internet Archive
summary
The Internet Archive stores about 25 petabytes of content including 482 billion web captures and adds roughly 2 billion captures per week; storage uses custom server bricks (36 disk slots, thousands of shingled drives) and four sharded indices (all, delta full, delta delta plus Redis for immediate “save page now” playback). A prototype Wayback search indexes HTML into Elasticsearch but takes ~100 days on current infra; tools include a Python client (try pip install internetarchive), mass upload/download support, and a book date-entity visualization.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:32Background
- 3:41Wayback Machine
- 5:58Archive
- 8:42Luxembourg
- 11:08Big Data
- 13:37sharding
- 14:43bots
- 15:08Greenfield
- 15:30Playback
- 15:55Two pieces to this
- 16:18Organizational challenges
- 16:45Building a search engine
- 19:18Elastic Search
- 19:38Help in the Environment
- 22:19Bad Search Engine
- 22:49Scaling
- 23:43API
- 26:17Command Line Tools
- 34:39Comments
- 36:18Takeaways
- 39:06Questions
related talks
41mAddressing the Challenges of Public Web Data
26mSaving Data to the DWeb
37mWikimedia Content API: A Cassandra Use-case
01h 19mThe Future of the Past: Challenges in Archival Storage
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
14mHarnessing Common Crawl for AI and ML Applications
40mWikimedia Content API: A Cassandra Use-case #bbuzz
48mHow to Analyze Terabytes of Data from GitHub Archive at High Speed