Messing up Your NUMA Topology with CXL
summary
CXL devices create extra NUMA nodes, the BIOS HMAT often lists many more nodes than actually usable, and measured behavior disagrees with tables: qemu emulated CXL demo and real hardware show CXL node latency around 300 ns (600 ns off-socket) while bandwidth drops to about 52 GB/s (26 GB/s non-optimal), far from HMAT’s advertised 256 GB/s. Kernel memory tiering/demotion based only on latency misplaces hot memory; a latency+bandwidth metric or runtime I/O scheduling is needed.
AI-written summary · about
chapters (10)
- 0:00Introduction to NUMA
- 0:45Architecture evolution
- 6:48NUMA basics and kernels
- 9:04Modern latency challenges
- 10:16Programming model shift
- 10:36I/O and NUMA awareness
- 15:56CXL and memory tiering
- 23:06Real-world CXL benchmarks
- 32:27I/O latency complications
- 38:34Discussion and conclusion
related talks
26mCXL Memory Tiering for heterogenous computing
37mMeta’s CXL Journey and Learnings in Linux
43mExpanding the Memory Limits with CXL Software Stack
39mCompute Express Link(CXL), the Next Generation Interconnect - Overview and the Status of Linux
31mHMSDK: Enhancing Software Support for CXL Memory
29mCXL HDM-DB support for Linux
28mCXL Type-3 device discovery, configuration in firmware and prepare ACPI tables...
38mMaximizing Heterogeneous Memory Bandwidth in Multi-Socket Systems