High Density VMs in OpenShift: The Journey to Native K8s Swap
summary
Describes a concrete engineering path: a custom wasp-agent that uses OCI runtime hooks and the OCI update binary to work around systemd cgroup reconciliation, plus a node-local controller to enforce Kubernetes-style limited swap; leverages FPR and KSM and puts swap on dedicated NVMe swap devices. Tests run on 64-core, 192 GB nodes with 6 vCPU/20 GB VMs show stable operation and no OOMs up to about 12.5% overcommit with acceptable performance.
AI-written summary · about
chapters (23)
- 0:00High-density VMs in OpenShift
- 1:07Why higher VM density matters
- 2:37KubeVirt memory requests
- 3:42KSM and FPR memory overcommit
- 5:30Kubernetes swap challenges
- 6:58Wasp agent for swap support
- 9:03Systemd vs Cgroup memory
- 10:48Short-lived containers race
- 11:50OCI hooks vs NRI choice
- 12:32Swap implementation limits
- 14:36Kubelet evictions and swap
- 16:40Kernel tuning vs CgroupFS
- 17:42Risks of beta Kubernetes swap
- 18:24Rebasing on GA Kubernetes swap
- 19:52Services excluded from swap
- 20:14Swap storage provisioning
- 21:58Mutable vs immutable OS swap
- 23:02Machine Config Operator for swap
- 23:44Safeguards for Kubernetes swap
- 24:26Future Kubernetes swap improvements
- 24:47Performance tests with VM density
- 26:58Journey to native Kubernetes swap
- 27:39Q&A on swap limits
related talks
14mScaleSwap: A Scalable OS Swap System for All-Flash Swap Arrays
29mOCI Hooks: Enhancing Kubernetes Runtime Management
43mMoving the Unmovable: migrating from VMs to K8s
20mBenchmarking HPC workloads on OpenShift
27mPods on Steroids - A Case Study of Migrating Flipkart.com to K8s on Bare Metal
44mPerformance and Scalability Tuning Kubernetes for OpenShift and Docker
15mBenefits of Taking the Less Traveled Road with Containers Infrastructure
41mOpenShift: a Kubernetes Story in an Enterprise Environment