~/talk

BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching

on YouTube 14m
Talkat OSDI '25 · 2025-07-07..07-09· [machine-learning, distributed-systems]

speakers

name:Dingyan Zhang
links:none on file
name:Haotian Wang
links:none on file
name:Yang Liu
linkedin:yang-liu-796211b1
name:Xingda Wei
links:none on file
name:Yizhou Shan
links:none on file
name:Rong Chen
links:none on file
name:Haibo Chen
x:HaiboChen

resources

video:https://youtu.be/pS5ofuHOVes
slides:none on file

summary

System claims live, fast autoscaling by avoiding full model caching with O(1) host caching and loading parameters over compute fabrics (RDMA) using a network-optimized multicast plan that greedily uses fast links and sharded transmission. Scaling moves from instance-level to layer-level offload, enabling cooperative execution where new GPUs borrow layers from active GPUs. Implementation in Rust/C++ reports up to 94% lower tail latency vs ServerlessLLM and 49% reduced GPU serving time versus non-autoscaling baselines.

AI-written summary · about

chapters (8)

  1. 0:00Introduction to BlitzScale and autoscaling challenges
  2. 2:03Instance startup overhead
  3. 4:57Caching limitations
  4. 6:46Multicast model loading
  5. 9:16Model sharding for bandwidth
  6. 11:01Live autoscaling with cooperation
  7. 13:07Implementation and results
  8. 13:50Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.