From Cold Start To Warp Speed: Triton Kernel Caching With OCI Container Images
summary
Proposes packaging Triton-generated LLVM kernel caches into signed OCI container images to prepopulate nodes and bypass JIT compilation, implemented as MCU (MCM + MCV) for index/prune/warm/sign and a GKM operator with a CSI driver to extract caches on nodes. Demo prototype shows a cached VLM pod startup drop from 115s to 61.5s and torch compile time from 56s to 11.93s (roughly 2x total startup speedup). Uses cosign, Docker/Podman registries and works in Kubernetes.
AI-written summary · about
chapters (10)
- 0:00Triton kernel caching intro
- 0:44What is Triton and caching
- 2:07JIT compilation in Kubernetes
- 3:14Model Cache Utils and GKM
- 3:57Model Cache Manager deep dive
- 9:12Model Cache Vault for OCI
- 11:17GPU Kernel Manager operator
- 14:51Live demo setup
- 18:04Benchmarking cached vs uncached
- 19:51Q&A
related talks
40mNode-Level Optimization in Large Cluster Powered by Containerd Plugin
30mBeyond Orchestration: The Cloud Native Runtimes Ecosystem for Performance and Security
25mStreamlined Efficiency: Unshackling Kubernetes Image Volumes for Rapid AI Model...
44mThe Containerization of the Operating System: Exploring Bootc and the Future of Linux Deployment
37mCRI: The Second Boom of Container Runtimes - Harry Zhang & Xu Wang, HyperHQ
32mOptimizing Application Performance on Kubernetes
01h 04mCommon-sense acceleration of your MLOC build
49mContainers All the Way Down!