Fast And Portable LLM Inference With WebAssembly And Rust
summary
Claims a zero python dependency LLM stack built with Rust and WasmEdge that runs the same binary on a 16GB M1 laptop and on an Nvidia cloud GPU without recompilation. Demonstrations show a lightweight runtime and app under 30 megabytes, Llama3 8B loading locally and serving an OpenAI‑style API, a quadrant vector DB knowledge base, and generation speeds around 24 tokens per second. Over 10,000 fine‑tuned models on Hugging Face are supported and the demos run successfully.
AI-written summary · about
chapters (14)
- 0:00WebAssembly and LLM inference intro
- 4:07WebAssembly’s 2024 killer apps
- 6:13LLM demo on MacBook
- 11:09Avoid cloud LLMs like OpenAI
- 12:56WebAssembly benefits for LLM deployment
- 15:27Demo: porting to Nvidia GPU
- 19:03WebAssembly for LLM apps
- 26:47Rust vs Python for LLMs
- 27:53Rust LLM inference code
- 30:01Demo: RAG app in Rust
- 36:49Switch models in one command
- 41:57Fine-tuned LLMs ecosystem
- 46:51Building local LLM knowledge bases
- 47:58Q&A: model formats & privacy
related talks
33mEfficient and Cross-Platform LLM Inference in the Heterogenous Cloud
25mLeveraging Wasm for Portable AI Inference Across GPUs, CPUs, OS & Cloud-Nativ...
30mWrite One Run Anywhere, But for GPUs
27mWasm Is Becoming the Runtime for LLMs
43mBuilding Serverless AI Workflows with Wasm+Rust
41mCreating Versatile AI Agents Through Wasm+Rust
40mWebAssembly for AI Infra : A Lightweight, Fast, and Secure Alternative Approach
40mOpen Source LLMs in the Cloud: Scalable Solutions