Maximising Data Storage Efficiency with Ceph as a Caching Layer for Tape Archiving
summary
A working prototype uses ceph librados+rgw as an S3 interface, an NVMe cache and a separate Blue data manager that directs a blob-store to move objects onto tape; archival writes truncate RADOS object payloads to zero while preserving metadata and truncate rados objects and write blobs directly to tape (no LTFS). A PostgreSQL index tracks which blob lives on which tape. Demo writes (1–20 MB), archives, restores and checksums successfully; hardware target is an IBM TS4500 4,600 slots ~82PB library with 10 drives for parallelism.
AI-written summary · about
related talks
14mCost-efficient Archive Cloud Storage with Tape: Design and Deployment
36mRapid Processing of NASA Satellite Data Stored with Ceph
36mMigrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
30mProject Caerus: Compute Storage Coordination
17mBoosting Cost Efficiency by Cost Aware Architecture
28mA New Way to Build A Ceph powered Data Lake for Big Data
26mPetabyte Scale Datalake Table Management with Ray, Arrow, Parquet, and S3
27mBuilding a PostgreSQL data warehouse