Workshop: Efficient and Portable AI / LLM Inference on the Edge Cloud
summary
Shows a working pipeline that swaps heavy containers for a tiny Wasm binary and runtime, with wasm runtime <20MB vs PyTorch images ~3GB and C++ toolchains ~300MB. Uses Lage + Wasm to run Llama2/3 and vision stacks (YOLO, Mediapipe), launches a local web chat UI and a RAG pipeline with Kon vector DB and ganet, demonstrating retrieval of WWDC content. Reports concrete performance: Llama 3 8B at 427 tps on M2, and scaling notes like 70B Q5 ≈60GB, H100 ~20–50 tps; demos run successfully on laptop.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:06Open source lightweight LLMs
- 7:35Web chat UI with Wasm
- 16:16Demo one: API compatibility
- 16:38Retrieval-augmented generation
- 20:14Demo two: RAG with Ganet
- 39:32Debugging RAG workflow
- 43:01Lage with Lobby Chat UI
- 46:52Container workflows
- 47:35Q&A
related talks
25mLeveraging Wasm for Portable AI Inference Across GPUs, CPUs, OS & Cloud-Nativ...
55mFast And Portable LLM Inference With WebAssembly And Rust
10mLightning Talk: Scaling AI With Wasm and Edge Computing
37mGetting Started with AI and WebAssembly
40mOpen Source LLMs in the Cloud: Scalable Solutions
27mWasm Is Becoming the Runtime for LLMs
40mWebAssembly for AI Infra : A Lightweight, Fast, and Secure Alternative Approach
33mReady, Set, Go: WASM-Powered Containers Taking Your AI To New He...