Running Intelligence at the Edge: Web AI in Action
summary
Shows a live flow converting PyTorch/TensorFlow models to ONNX, applying quantization/pruning, and serving a 2–4 billion parameter LLM in-browser with weights chunked from a CDN and cached in IndexedDB. Demonstrates an offline response from a WebGPU + WebAssembly runtime using WebLLM (and ONNX.js for non-LLM tasks), reports 4‑bit quantized Gem/Gemma models running token-by-token, and measures eliminated API calls, lower latency, and reduced server GPU costs.
AI-written summary · about
chapters (6)
- 0:00On-device AI in browsers
- 3:12Why on-device AI matters
- 8:12Browser AI runtimes
- 11:45Web LM vs ONNX
- 12:29Live demo: Web LM
- 16:08Browser AI limits & future
related talks
41mEscaping the Cloud: High-Performance AI in your Browser
46mFrom ML to LLM: on-device AI in the browser
46mThe Ultimate Guide to Browser-Based AI: Privacy-First, Offline-Ready and Future-Proof
18mWeb AI on next generation AI PCs
26mDeploying PyTorch Models To the Browser and Beyond With Transformers.js
20mFrom ML to LLM: On-Device AI in the Browser
51mMaking neural networks run in browser with ONNX
13mLiteRT.js, Google’s high performance WebAI runtime