Exabyte Data Storage
summary
Claims an exabyte-capable archive with 1.4 exabyte container prototypes built from racks of mini storage pots (16×20 TB each) that hold four years of telescope data, using three-way replication across two continents and checksums plus layered metadata ("data onion") for bit-rot detection and repair. Offers transaction IDs, access tokens, unlimited versioning, continent-aware placement and stable Swift/REST APIs. Replaces legacy Java with Swift components, runs on Apple hardware, and reports 20% power use vs equivalent Dell in tests.
AI-written summary · about
chapters (7)
- 0:00Introduction to exabyte storage
- 1:11Why build exabyte systems
- 6:10Design goals and requirements
- 12:37Implementation and encryption
- 19:43Custom hardware for observatories
- 23:03Current project status
- 25:11Call for collaboration
related talks
39mChallenges of Building Blazing Fast Data APIs
57mFast encrypted backups with Rust
13m"Blazing-Fast Magic Beans"
22mTransparent proofs of data records in blockchain
44mMemory mapped storage
11mAdapting TiKV for Cloud Storage
19m"Embracing Monorepo and LLM Evolution"
27mCrimson: A New Ceph OSD for the Age of Persistent Memory and Fast NVMe Storage