P2P Distributed LLM Inference Across Browsers With llama.cpp And WebRTC
summary
Shows a working LLMlet implementation that compiles llama.cpp to WebAssembly, patches the llama.cpp RPC backend to run over WebRTC, and shards models at layer granularity so a browser cluster can run a QN 3.59B model. Demo runs across four tabs, uses PRJS signaling, caches model files in IndexedDB, exposes JS function calling and an emulated Linux terminal via QEMU, and fits a single-tab heap to 4 GB; inference speed is ~3 tokens/second and parallelism and TURN relays are not yet supported.
AI-written summary · about
chapters (10)
- 0:00Introduction to LLMlet
- 2:06Browser inference benefits
- 4:29LLMlet architecture
- 6:06P2P distributed inference
- 8:41Function calling capabilities
- 10:24VS Code LLMlet extension
- 11:51Live demo and terminal emulation
- 17:14Deep dive into project components
- 22:57Challenges and related work
- 25:17Summary and Q&A
related talks
16mReact + WebLLM: Building Intelligent Applications with In-Browser Language Models
40mOpen Source LLMs in the Cloud: Scalable Solutions
34mPrivacy First: Building LLM-Powered Web Apps with client side WASM
17mWebLLM: A high-performance in-browser LLM Inference engine
17mRunning WebLLM in the Browser on RISC-V Toward Lightweight, Local AI Experiences
45mEverything You Need to Know About Running LLMs Locally
47mOwn Your LLM: Running Generative and Agentic AI Locally
31mFrom ML to LLM on device AI in the browser