Tessera: A Holistic Pipeline Parallelism Framework for Trillion-Parameter Heterogeneous
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
Tessera is a pipeline-parallel framework that profiles fine-grained, per-pair overlap schedules and uses those measured post-overlap costs to cut stages, then fills routing-induced idle slots at runtime. It addresses >50% all-to-all cost in Qwen-3 MoE by combining an overlap scheduler, an overlap-aware partitioner and a dynamic bubble optimizer that injects movable gradient tasks. Deployed at 4,096–12,288 GPUs, it raises throughput 20–33% (up to 39% MFU) and achieves up to 1.24× MFU vs Megatron-Core MoE.
AI-written summary · about
chapters (9)
- 0:00Tessera pipeline parallelism
- 1:01Pipeline and expert parallelism
- 2:48Heterogeneous MoE bottlenecks
- 4:14Why traditional solutions fail
- 6:44Breaking planning cycle
- 8:31Handling MoE runtime variations
- 11:01Dynamic execution engine
- 12:29Production performance gains
- 15:44Summary and future
related talks
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
20mScheduler and Sharding Considerations for Network Efficiency
11mQiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a...
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
15mScaling the Next Paradigm of Heterogeneous Intelligence
15mPipeThreader: Software-Defined Pipelining for Efficient DNN Execution
20mMetis: Fast Automatic Distributed Training on Heterogeneous GPUs
22mThemis: Fair and Efficient GPU Cluster Scheduling