Optimizing Ceph Object Storage for Analytics and AL/ML Workloads - Overview and Future Plans
summary
Ceph demonstrates concrete scale and performance: tests hit 10 billion objects on six nodes and a single-bucket run holds 250 million objects, while a 10-node disk cluster sustains ~80 GB/s aggregated throughput; real deployments need ~1500 simultaneous TCP connections. Prototypes exist for a converged object cache (D4N) plus the Careers prefetcher that materializes data into cache before workers run, object-packing for small-write efficiency, zipper filters and alternate backends, and a Quick multi-streaming front end — most components are working as prototypes and are being fitted for upstream and product use.
AI-written summary · about
chapters (5)
- 0:00Introduction and background
- 2:35Modern analytics storage landscape
- 10:05Research collaboration with Massachusetts Open Cloud
- 22:17Integrating D4N and cache technologies
- 27:15Q&A on tool adoption and compatibility
related talks
36mRapid Processing of NASA Satellite Data Stored with Ceph
36mMigrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
28mA New Way to Build A Ceph powered Data Lake for Big Data
22mWrite-Through Object Cache for Fleets of Global Edge Clusters
30mProject Caerus: Compute Storage Coordination
14mBuilding a high-performance analytics ecosystem on Amazon EKS
14mFalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
39mUnlock the potential of your data: Optimize your data store architecture...