AI, ML, and the Ceph Advantage: Scalable Storage for Smarter Workflows
summary
Ceph serves as object block and file storage for AI/ML pipelines, with live multi‑site replication, scrubbing to detect silent disk failure, and automation to raise servers‑per‑admin far above the typical ~100. Tradeoffs are concrete: erasure coding 3+2 = 60% efficiency versus replica‑3 for higher performance; hardware choices matter—NVMe for performance, spinning disks for capacity—and networking ranges from 25G to 800G. A tested example: 5‑node 100‑disk 20TB ≈6PB usable. Ops guidance: avoid Ceph in Kubernetes and run bare metal with observability and right‑sized tooling.
AI-written summary · about
chapters (24)
- 0:00AI ML storage needs
- 1:12Sardina Systems fishOS
- 2:16Storage for AI workloads
- 3:22Ceph capabilities
- 5:14Separate compute and storage
- 6:21AI edge storage scalability
- 7:47AI pipeline data ingestion
- 9:14Ceph replication and protection
- 9:56Automate storage scaling
- 11:00Ceph performance myths
- 12:04Ceph hardware considerations
- 13:10Ceph deployment complexity
- 14:37Ceph erasure coding
- 15:43Hardware failures and integrity
- 16:47Ceph failure handling
- 17:31Ceph hardware specs
- 18:36Tuning Ceph for AI
- 19:58Storage pools for workloads
- 20:42Bare metal Ceph overhead
- 21:23Ceph observability
- 22:32Cost versus performance tiers
- 23:38Key takeaways for AI storage
- 24:42Ceph is not slow
- 26:50Q&A hardware investment
related talks
36mMigrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
43mCeph and Flash
14mArchitecting Cloud Storage for AI Native Applications
34mEvaluating the Performance and Power Efficiency of Ceph Storage Configurations
12mOptimizing AI Data Infrastructure with F5 and
31mRook: Intro and Deep Dive With Ceph
29mCeph Object Storage Overview, Capabilities and Future Plans
21mAIaaS with Dell AI ready infrastructure