Messing up Your NUMA Topology with CXL
summary
CXL memory creates unexpected NUMA behavior: HMAT entries claim many small nodes but real systems show six usable nodes; measured latencies are ~300 ns optimal and ~600 ns non‑optimal while BIOS HMAT promises far lower numbers, and bandwidth to CXL is much lower than expected (about 52 GB/s vs 26 GB/s in optimal/non‑optimal runs, versus a claimed 256 GB/s). Experiments use a qemu emulated CXL and a Granite Rapids prototype with a modified RAM driver; current Linux tiering uses latency-only distances and needs bandwidth-aware placement or runtime IO/placement scheduling.
AI-written summary · about
chapters (10)
- 0:00Introduction to NUMA
- 0:45Architecture evolution
- 6:48NUMA basics and kernels
- 9:04Modern latency challenges
- 10:16Programming model shift
- 10:36I/O and NUMA awareness
- 15:56CXL and memory tiering
- 23:06Real-world CXL benchmarks
- 32:27I/O latency complications
- 38:34Discussion and conclusion
related talks
12mDirect Access, High-Performance Memory Disaggregation with DirectCXL
11mXerxes: Extensive Exploration of Scalable Hardware Systems with CXL-Based Simulation...
45mBeyond Machine Boundaries: Experience with Memory Disaggregation
17mCylon: Fast and Accurate Full-System Emulation of CXL-SSDs
46mA Complete Re-Think for Memory Config...
23mBreaking the limits of x86
16mOneSidedMW: Managing Disaggregated Memory Efficiently, Flexibly, and Securely with RNIC
23mTeRM: Extending RDMA-Attached Memory with SSD