OminiX - Unified Acceleration Framework for Both LLM and SD GenAl Models on the Edge
summary
OminiX is an open-source runtime that unifies acceleration for Stable Diffusion and LLMs (LLaMA2 7B/13B, Mistral 7B) with on-device finetuning and a common operator set, claiming real-time edge inference and removes ONNX 2GB limit. Demonstrations show Stable Diffusion under 4s on NVIDIA, about 2–3s on iPhone and ~6s on Android, Llama 7B at ~70–80 tokens/sec on an M1 (110–120 expected on M2/M3), and planned release of a fully open 7B model plus codegen for Android/iPhone/Mac and 20–30T training data and 5–6x size reduction via quant/prune.
AI-written summary · about
chapters (10)
- 0:00Introduction to OminiX
- 2:53User-centric AI deployment needs
- 6:51Limitations of existing compilers
- 11:14Llama CPP and its shortcomings
- 13:24OminiX framework goals
- 16:59Stable Diffusion acceleration
- 22:45Cost and performance benefits
- 24:34Live demo of edge AI
- 26:22Open-source large language model
- 30:49Future of OminiX and community
related talks
23mLLMs on the Edge: The Future of On-Device Intelligence
02mInferentia2
45mDeliver AI-powered experiences across cloud and edge with Windows
35mOn‑Device LLMs on Android Devices 2025
15mBuilding Gen-AI experiences using LLM and LVM
39mMoxin: The All-In-One Tool to Run LLMs on Your Personal Device
23mWhy MLX
37mDeep Learning Neural Network Acceleration at the Edge