Combining Buffered I/O and Direct I/O in Distributed File Systems

on YouTube ↗22m

" summary

Implements a dynamic IO-mode chooser inside Lustre that switches between buffered and direct I/O based on IO size, lock contention, and memory pressure; adds unaligned direct IO handling, server-side writeback, cross-file batching and delayed allocation. Evaluated on NVMe-backed servers with up to 32 clients, IO500 microbenchmarks and a 512-process NE5000 CFD run (≈600 GB, 130k requests switched), yielding up to 3× throughput vs vanilla Lustre and up to 13× vs other systems; code is opt-in and landing in Lustre 2.16.

AI-written summary · about

" chapters (22)

1. 0:00Buffered vs direct I/O intro
2. 1:00Buffered vs direct I/O performance
3. 2:47Challenges and benefits of direct I/O adoption
4. 3:29Page cache cost in buffered I/O
5. 4:31Lustre architecture overview
6. 5:59When buffered and direct I/O perform best
7. 7:02Goals for mixed I/O
8. 7:45Unaligned direct I/O in Lustre
9. 8:07Client I/O mode decision
10. 9:54Lock contention impact on I/O
11. 10:38Memory pressure effects
12. 10:59Server-side writeback
13. 11:41Batch buffered writes
14. 12:24Delayed allocation benefits
15. 13:05Consistency with mixed I/O
16. 13:49Auto I/O config
17. 14:10Evaluation setup
18. 15:15Single process I/O results
19. 16:00MDTest small file analysis
20. 17:25IOR strided I/O performance
21. 18:29NEK5000 HPC results
22. 19:55Conclusion

" related talks

• 17mλ-IO: A Unified IO Stack for Computational StorageJiwu Shu, Zhe Yang, Youyou Lu, Xiaojian Liao, Junru Li, Youmin Chen, Siyu He · FAST '23: 21st USENIX Conference on File and Storage Technologies
• 47mExploring Linux Memory Usage and IO Performance in Cloud Native ApplicationsFrits Hoogland · Devoxx France 2022
• 16mWhen DDIO Meets Page Coloring: Revisiting DDIO Performance with SepiaQizhe Cai, Changwoo Song, Sanghyun Kim, Jinhyeok Oh, Joonsung Kim, Jaehyun Hwang · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)
• 01h 01mFile I/O for Game Developers: Past, Present, and Future with C++Guy Davidson · CppCon 2023
• 19mAsynchronous I/O Stack: A Low-latency Kernel I/O Stack for Ultra-Low Latency SSDsJinkyu Jeong, Tae Jun Ham, Jae W. Lee, Gyusun Lee, Seokha Shin, Wonsuk Song · USENIX ATC '19
• 19mBest I/O is No I/OFrank Mashraqi · Velocity 2009
• 36mIntroducing the libblkio High-performance Block I/O APIStefan Hajnoczi, Alberto Faria · KVM Forum 2022
• 19mSpeeding Up VM’s I/O Sharing Host's io_uring Queues With GuestsStefano Garzarella · KVM Forum 2020