Memory-Efficient LLM Inference on Edge Devices With NNTrainer
summary
NNTrainer (github.com/nnstreamer/nntrainer) repurposes its device training schedulers to run huge LLMs by combining memory-efficient tensor scheduling with a fine-grained flash storage utilization that asynchronously preloads weights. The system demonstrates running 20B–30B MoE models on a Galaxy S25 class device using 4‑bit quantization and on‑demand weight loading, dropping peak RAM from ~20 GB to runs 30B models with 1.6 GB while keeping reasonable latency and offering prompt caching for repeated system prompts.
AI-written summary · about
chapters (7)
- 0:00Introduction to NN Trainer and on-device AI
- 1:45On-device LLM challenges
- 8:29NNTrainer’s memory framework
- 15:20Running 30B models
- 21:53Memory-efficient LLM demos
- 24:21NNTrainer’s full capabilities
- 25:24Conclusion
related talks
17mCLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
23mLLMs on the Edge: The Future of On-Device Intelligence
35mOn‑Device LLMs on Android Devices 2025
30mUnderstanding the end-to-end LLM training and inference pipeline
21mThings I learned running neural networks on microcontrollers
20mGAMING: Benchmarking Sustainable LLM Deployment on Edge Devices
38mOptimizing LLM Inference for the Rest of Us