PMEM-backed Repositories for more high-performance Apache NiFi
summary
Proposal adds Content, FlowFile, and Provenance Repositories backed by persistent memory and implements access via Filesystem DAX and PMDK, modifying NiFi to perform byte-addressable load/store I/O instead of block writes. Built prototypes run on Optane PMem in App Direct Mode and replace disk I/O paths; performance tests report substantially lower commit latency and much higher throughput for large IoT flows, demonstrating the approach works and can be applied to other data systems like Kafka.
AI-written summary · about