Write One Run Anywhere, But for GPUs
summary
LlamaEdge compiles Rust (or C/C++/Go) apps to wasm bytecode and runs them on the LlamaEdge runtime built atop WasmEdge, routing WASI calls to native backends like llama.cpp, TensorRT, MLX, PyTorch, TensorFlow, candle and burn.rs. Demoed apps include a chat, an OpenAI‑compatible API server, and an RAG agent using Qdrant; default GPU offload is 100 layers (Llama2/3 ≈33 layers so fully offloaded), wasm apps shrink to ~5–6 MB and run at near‑native speed on devices from Raspberry Pi4 to CUDA/Metal/Qualcomm hardware via WASI-NN, and the examples run successfully.
AI-written summary · about
chapters (7)
- 0:00Introduction to WasmEdge
- 1:36Basic chat application demo
- 4:30Software stack architecture
- 11:00API server implementation
- 18:10Hardware and deployment
- 22:16RAG application demonstration
- 26:43Container integration
related talks
33mEfficient and Cross-Platform LLM Inference in the Heterogenous Cloud
55mFast And Portable LLM Inference With WebAssembly And Rust
25mLeveraging Wasm for Portable AI Inference Across GPUs, CPUs, OS & Cloud-Nativ...
27mWasm Is Becoming the Runtime for LLMs
41mCreating Versatile AI Agents Through Wasm+Rust
43mBuilding Serverless AI Workflows with Wasm+Rust
40mOpen Source LLMs in the Cloud: Scalable Solutions
48mWorkshop: Efficient and Portable AI / LLM Inference on the Edge Cloud