Enabling Near-Data Processing in Ceph

on YouTube ↗20m

" summary

Adds an aggregate module that groups same-pool small objects into a single stripe-aligned aggregate object so no original object is split before erasure coding, and extends OSDs with a near-data processing module and new read requests to run operations (regex, DB-style filters, third-party libs) on the OSTs. Tests on six OSTs (4 data + 2 parity) reading 200 objects (~204 MB total) show network traffic dropping from about 600MB to 300MB, WIP but functional for read/filter workloads.

AI-written summary · about

" chapters (9)

1. 0:00Introduction to near-data processing
2. 1:08What is near-data processing
3. 2:34Why near-data processing is hard
4. 4:25Ceph pools and Erasure coding
5. 5:50Enabling near-data processing in Erasure code pools
6. 9:48Near-data processing in OSDs
7. 13:02Test setup and results
8. 17:29Future work and summary
9. 18:15Q&A