KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for M...
summary
KTransformers is an inference system for Mixture-of-Experts (MoE) models that targets CPU/GPU hybrid execution by using CPU/DRAM for large memory and GPU/VRAM for bandwidth-heavy work. It implements AMX-optimized CPU kernels and an asynchronous CPU–GPU task scheduler to cut scheduling overhead, reporting 4.62–19.74× speedups for prefilling and 1.25–4.09× for decoding versus prior systems.
AI-written summary · about
related talks
23mThe Race to the Bottom: Low Latency in the age of the Transformer
14mKairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing
42mAnima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor, CalTech
01h 18mOptimizing LLM Training and Inference Performance on GPUs
15mUSHER: Holistic Interference Avoidance for Resource Optimized ML Inference
36mTransformers in SRE Land: Evolving to Manage AI Infrastructure
15mScaling the Next Paradigm of Heterogeneous Intelligence
11mQiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a...