Storing your data in kernel space: an excellent bad idea!
summary
Claim: put application data into kernel buffers by creating lots of pipes and managing offsets instead of using user memory. Tools: ordinary pipes (default ~4 KB, configurable up to 1 MB), many file descriptors and small helper processes to cycle ends. Numbers: per‑user big‑pipe quota (~64 MB) and experiments that push allocations into hundreds of megabytes or multiple gigabytes. Result: a toy program that allocates thousands of pipes works enough to store data but eventually trips resource limits and provokes the oom killer.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:14two kinds of code
- 2:48kernel code
- 3:07bookkeeping
- 3:22user code can't access kernel memory
- 6:04a kernel buffer the user has control over
- 6:38how can we get a random access buffer
- 9:1364K file descriptors
- 13:08total fle descriptor limit
- 13:22memory limit
related talks
36mThe Perils of Large Files
21mLessons Learned Building a Production Memory-Overcommit Solution
45mData Placement in Linux: Evolving Block and File I/O
52mChanging shared_buffers on the fly
37mLarge Block I/O for Linux
33mThe Challenge of Storing Small Objects on a Large Scale
10mExtMem: Enabling Application-Aware Virtual Memory Management for Data-Intensive...
30mPostgreSQL and Linux Kernel interactions