From ML to LLM: on-device AI in the browser
summary
Browser runs ML end to end: TensorFlow.js and WebAssembly prototype shows CPU at ~5 FPS, WebGL ~13 FPS and a webGPU backend at ~30 FPS for face and hand tracking demos. Transformers.js + ONNX/ONNX Runtime and Apache TVM compile LLMs so a Gemma 2B baseline (2B params ~30 GB) can be quantized to 1.4GB and served as Gemma 2B in-browser. Examples include real‑time captions, MD editor at ~27–30 tokens/sec, and ask‑my‑PDF with 81 ms parsing, 1.25 s vectorizing, 61 ms retrieval, and <10 s answers.
AI-written summary · about
chapters (11)
- 0:00Introduction to web AI
- 3:07TensorFlow.js and hardware
- 6:05Interactive browser projects
- 9:13Web Neural Network API
- 12:30Custom model training
- 15:15Real-time speech transcription
- 19:29LLMs in the browser
- 26:39LLM applications
- 29:53RAG for browser documents
- 36:44Future of web AI
- 44:01Conclusion and Q&A
related talks
20mFrom ML to LLM: On-Device AI in the Browser
31mFrom ML to LLM on device AI in the browser
19mML at Client Side
22mBoring but Effective: WebAI That Ships
41mEscaping the Cloud: High-Performance AI in your Browser
17mRunning Intelligence at the Edge: Web AI in Action
34mPrivacy First: Building LLM-Powered Web Apps with client side WASM
17mWebLLM: A high-performance in-browser LLM Inference engine