Better Indexing and Faster Retrievals for Decentralized Hot Storage
summary
Describes a concrete system that replaces a 52‑node Cluster babysit setup with cars‑at‑rest via HTTP PUT to simple object storage, adds a compact index that records per‑block offsets and lengths and a shard SID so blocks can be byte‑range fetched, and uses IPNI metadata entries (index CID + URL) published via synthetic hashes to map root CIDs to car locations. Claims this makes uploads transactional, retrievals deterministic and fast, and lets lightweight retrieval nodes assemble DAGs without running full IPFS nodes.
AI-written summary · about
chapters (5)
- 0:00Introduction to decentralized indexing
- 3:53Push-based storage with CAR files
- 8:54Indexing CAR files for retrieval
- 13:35Decentralizing with IPNI
- 18:35Storacha network workflow
related talks
38mLight and Adaptive Indexing for Immutable Databases
52mTaking Storage for a Ride
33mNext Gen of Datastores: Hot Queries, Cold Storage
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
28mAerospike's Journey Through Innovations, Sustainability, and Roadmap
32mLow Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...
36mHot Swap Your Datastore: A Practical Approach and Lessons Learned
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data