Wasm Is Becoming the Runtime for LLMs
summary
Claims that Wasm replaces Python for LLM apps, showing a Rust-compiled single 2MB wasm binary that loads a 5 GB Llama-2 model and runs inference locally via WASI NN using GGML/TensorFlow/PyTorch backends, streaming tokens from the GPU for interactive speed. Demonstrates an OpenAI-compatible JSON server on port 880, integration with containerd/runwasi for Docker/Kubernetes deployment, and media-pipe image models; startup stays slow due to model load but mmap and wasi host-API optimizations are in progress.
AI-written summary · about
chapters (10)
- 0:00Wasm’s evolving role
- 2:12Custom LLM for Wasm
- 3:55AI inference with Wasm
- 7:50Wasm and neural networks
- 10:01Rust-based Wasm AI app
- 13:54Docker for Wasm AI
- 16:07Wasm beyond LLMs
- 18:15Python replaced by Wasm
- 20:24Wasm portability and performance
- 22:13Q&A
related talks
40mWebAssembly for AI Infra : A Lightweight, Fast, and Secure Alternative Approach
37mGetting Started with AI and WebAssembly
25mLeveraging Wasm for Portable AI Inference Across GPUs, CPUs, OS & Cloud-Nativ...
41mCreating Versatile AI Agents Through Wasm+Rust
55mFast And Portable LLM Inference With WebAssembly And Rust
33mEfficient and Cross-Platform LLM Inference in the Heterogenous Cloud
31mRunning AI workloads with WASM - is it production ready? [Panel]
30mWrite One Run Anywhere, But for GPUs