A journey through supporting VMs with dedicated CPUs on Kubernetes
summary
KubeVirt adds VMs to k8s by running QEMU inside containers; dedicated CPU pinning requires Guaranteed QoS and integer CPU requests. Early attempts either context-switch vCPU threads or waste an extra core with an X+1 core housekeeping cgroup. Final solution spawns an emulator container with X dedicated CPUs and moves only vCPU kernel threads into its cgroup, keeping management threads on shared CPUs; this avoids the extra core and most cgroup v2 limits but hits v2’s cgroup v2 thread subtree constraint and remains a work in progress.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:58Introduction to Kubevirt
- 3:45VMs with dedicated CPUs
- 4:48What are containers?
- 5:38Introduction to Cgroups
- 8:03Threads model in v2
- 8:58How is CPU allocation implemented in k8s?
- 9:53Kubernetes CPU allocation: requests
- 11:52Kubernetes and dedicated CPUs
- 12:29Introduction to Namespaces
- 13:01Sharing PID namespace in a Pod
- 13:40A word about VM (Kernel-based VM)
- 15:37st attempt to support dedicated CPUs
- 16:04Inside the compute container
- 18:33nd attempt: IsolateEmulatorThread
- 19:49rd attempt: housekeeping approach
- 20:45Problems with housekeeping approach
- 22:33th attempt: emulator container approach
- 23:463rd attempt: dedicated-cpu cgroup approach
- 28:34Summary & Takeaways
related talks
31mKubeVirt: What If Kubernetes Orchestrated Your VMs?
43mKubeVirt - Kubernetes, Virtualization and Your Future Data Center
19mKubevirt and the Cost of Containerizing VMs
43mMaking Running Virtual Machines in a Kubernetes Cluster a Mainstream Activity
30mKubeVirt: bringing virtual machines in a Kubernetes world
32mKubeVirt - making running virtual machines in a k8s cluster
38mKubeVirt Intro: Virtual Machine Management on Kubernetes
36mIt’s All for the Users. More Durable, Secure, and Pluggable. KubeVirt v0.53