Deploying and Operating CephFS for Scientific Applications at Fermilab
summary
Fermilab runs CephFS in production since spring 2024 after a spring 2023 rollout, using cephadm to bootstrap and manage a cluster with about 1.5 PB raw / 500+ OSDs on 25 storage nodes (half NVMe, half HDD), five MONs, five MDS (4 active + 1 standby) and three mgrs. Operations note: kernel client on most nodes, upgrades without downtime, MDS memory-leak bug fixed upstream, occasional disk/server failures tolerate rebuilds, scrubbing backlog ~10% PGs, and custom tools automate host reboots and hourly quota updates plus an auto-evict after 10m policy for stuck clients.
AI-written summary · about
chapters (16)
- 0:00Introduction to Fermilab
- 1:05Fermilab’s scientific experiments
- 3:34Data storage and analysis workflow
- 4:37Why choose CephFS
- 5:41CephFS cluster deployment timeline
- 6:25Cluster hardware and configuration
- 8:55Deployment challenges and solutions
- 10:01Operational challenges and fixes
- 11:45Security and compliance measures
- 12:50Performance tuning adjustments
- 13:52Custom monitoring and management tools
- 16:44Current and future use cases
- 18:08Object storage expansion plans
- 20:17Block device integration with dCache
- 21:20Summary and future outlook
- 21:40Q&A session
related talks
41mBeyond Particle Physics: The Impact of Ceph and OpenStack on CERN's Multi-D...
26mPetabyte Scale Object Storage Service Using Ceph in A Private Cloud
38mBringing a Ceph Based Enterprise Email System Into the Field
21mCeph for Public Cloud Workloads
05mThe Enterprise Ceph Spectrum
06mPractices of Ceph Object Storage in Public Cloud Services
15mThe CERN Cloud Infrastructure: 5 years in 15 minutes
45mAlmaLinux at CERN