JENGA: Enhancing LLM Long-Context Fine-tuning with Contextual Token Sparsity
summary
JENGA identifies and prunes redundant tokens during long-context fine-tuning to cut activation memory and speed up training, claiming up to 1.93× memory reduction and 1.36× speedup versus prior fine-tuning systems. It implements three concrete techniques: Token Elimination (block-wise attention scoring and layer thresholds), Pattern Prediction (lightweight Q/K predictors with elastic neuron pruning), and Kernel Optimization (permutation-free selection and segment-based peak cutting) and ships as an end-to-end system compatible with common LLM architectures.
AI-written summary · about
chapters (11)
- 0:00JENGA and long-context fine-tuning
- 1:21Long-context fine-tuning challenges
- 3:51Memory bottlenecks in activation
- 5:40Contextual token sparsity
- 7:07Challenges in token sparsity
- 8:33JENGA system design
- 9:36Token elimination algorithm
- 11:00Informative score predictors
- 12:23Permutation-free strategy
- 13:46Experimental results
- 14:49Conclusion
related talks
15mStop Stuffing Your LLM's Context Window
51mJlama: A Native Java LLM inference engine
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
38mOptimizing LLM Inference for the Rest of Us
24mLLM Inference Optimization From Simple To Sophisticated
01h 00mThe Frontier between Retrieval-augmented and Long-context Language Models
51mLong context modeling and generalization: two perspectives
52mReduce LLM calls with Vector Search Design Patterns