From ML to LLM: On-Device AI in the Browser
summary
Claims practical on-device LLMs using WebAssembly and WebGPU with TensorFlow.js and Transformers JS, showing face-landmark demo jumping from ~5 fps on CPU to >30 fps on GPU and text models like Gemma 2B quantized to 4‑bit (~1.4 GB) running in-browser. Measurements report 133 tokens/sec for input processing and 1.68 ms per generated token. Two shipped demos — MD edit (local Gemma generation) and ask my PDF (local vector DB retrieval) — work fast, keep data client-side, and reveal hardware limits on non‑M3 Macs.
AI-written summary · about
chapters (11)
- 0:00On-device AI in browser
- 2:13Speech recognition to TensorFlow.js
- 5:25Custom speech detection models
- 7:40Can large language models run in the browser
- 10:47LLMs with WebAssembly & WebGPU
- 11:30MD Edit demo with LLM
- 13:20LLM limits & retrieval augmentation
- 14:24Ask My PDF for private Q&A
- 16:33Hardware & future of on-device AI
- 18:39AI as web app enhancement
- 19:45Q&A
related talks
46mFrom ML to LLM: on-device AI in the browser
31mFrom ML to LLM on device AI in the browser
34mPrivacy First: Building LLM-Powered Web Apps with client side WASM
22mBoring but Effective: WebAI That Ships
19mML at Client Side
17mWebLLM: A high-performance in-browser LLM Inference engine
41mEscaping the Cloud: High-Performance AI in your Browser
17mRunning Intelligence at the Edge: Web AI in Action