Migrating AI/ML Data Pipelines for Real Time Streaming Applications from HDFS to Ceph
summary
Etraveligroup migrates 20 AI/ML streaming pipelines from HDFS to Ceph RGW on NVMe, testing two pipelines (50k and 20k records/min, 1 GB throughput, ~1s latency targets). Solutions include disabled C-states and IOMMU for latency, RDMA experiments, per-device queue depth 2, OSD bench tuning (baseline 21k → 160k), memory_target raised (4 GB → ~17 GB), RGW thread pool 1024 and 9 RGW instances. Result: millisecond response times and ~5,000 IOPS per minute from three OSD servers, stable production use.
AI-written summary · about
chapters (21)
- 0:00Introduction and company overview
- 1:03Problem definition and project scope
- 2:32Challenges with HDFS storage
- 4:41Why Ceph was chosen for migration
- 5:01High-level Ceph architecture
- 7:09Setting up the Ceph cluster
- 7:51Hardware tuning for low latency
- 8:54RDMA configuration and challenges
- 10:21Load testing methodology
- 11:26Benchmarking disk performance
- 12:07OSD profile and NVMe optimization
- 13:51Memory profiling and TCMalloc
- 14:34RADOS Gateway tuning
- 16:45RADOS Gateway instances scaling
- 18:08Data caching strategies in Ceph
- 19:32Key lessons learned
- 21:16Bucket sharding and performance impact
- 22:19Data security and access control
- 22:42Final benchmark results and outcomes
- 23:46Next steps and scaling plans
- 24:27Q&A session
related talks
31mAI, ML, and the Ceph Advantage: Scalable Storage for Smarter Workflows
35mBuilding and Scaling Data Pipelines for Data Engineering With Ceph Object Storage
36mOptimizing Ceph Object Storage for Analytics and AL/ML Workloads - Overview and Future Plans
36mRapid Processing of NASA Satellite Data Stored with Ceph
43mOptimizing Ceph IO for High Throughput Particle Physics Workflows
27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme
14mFrom Reservation to 1,600 GPUs: A Petabyte-Scale Ray Data Pipeline
36mAn almost perfect match: cloud native and scientific computing