Kairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
Kairox is a GPU–CPU hybrid inference system that dynamically moves FFN neuron groups at runtime to trade CPU compute for PCIe transfers, implemented in ~6,000 lines of C++/CUDA and evaluated on consumer PCs (PCIe3 12 GB, PCIe4 24 GB). It uses Live Pipeline prefetching to overlap copies, Temporal Activation Momentum to avoid one-off reloads, and an Adaptive Neuron Balancer that tunes reload aggression. End-to-end throughput improves up to 7.57× (max) and yields geomean speedups ~3.15×/3.93× over llama.cpp and ≈2.1× vs sparse baselines.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:22LLM inference on consumer PCs
- 2:48Static neuron placement limits
- 4:16Online neuron balancing insight
- 5:23Carrots system design
- 6:25Overlap IO with computation
- 7:52Group neurons for transfers
- 9:00Temporal activation momentum
- 10:06Adaptive balancing with lambda
- 11:09Experimental setup
- 11:54Main results
- 13:17Key takeaways
- 13:59Q&A
related talks
21mKTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for M...
19mOptimizing CNN Model Inference on CPUs
32mBeyond Round-Robin: GPU-Aware Load Balancing for LLM Inference in Kubernetes
23mThe Race to the Bottom: Low Latency in the age of the Transformer
26mAmazingly Fast and Incredibly Scalable Inference...
21mXorbits Inference: Model Serving Made Easy
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
18mDeploying ML solutions with low latency in Python