Exo: Run your own AI cluster at home
summary
exo is an open source library that aggregates everyday hardware (iPhone, iPad, Android, Macs, Linux) into a single inference cluster, demonstrated by running Llama 405B as a 200 GB split across two MacBooks using gRPC and tiny backend tooling; local networking or Tailscale joins remote nodes. Claim: large models run without datacenter GPUs; measured gains: single-device layer swapping hits ~0.3 tokens/sec while sharded multi-device runs reach ~12 tokens/sec. Project shows working demos, a local Siri-like agent, and 9.9k GitHub stars.
AI-written summary · about
chapters (9)
- 0:00Introduction to Exo
- 2:54Why Exo was built
- 7:08Running models on one device
- 11:23Scaling with multiple devices
- 17:01Key benefits of Exo
- 18:47Demo: Distributed cluster
- 21:47Demo: Agent on device
- 22:51Limitations and future
- 23:12Q&A and closing
related talks
47mOwn Your LLM: Running Generative and Agentic AI Locally
20mSelf-hosting AI LLMs - a beginners guide
24mPrivate AI at Home: Secure, Open-Source, and Fully Yours
31mOminiX - Unified Acceleration Framework for Both LLM and SD GenAl Models on the Edge
43mKeynote: Tiny Data Collectors, Vastly (...) & the Land of Tiny Challenges
37mUniversal AI: Execute Your Models Where Your Data (And Users) Are
17mFrom Agents to Action: Scaling Agentic AI with Ray on Intel Xeon
35mBuilding a 5-Exaflop Supercomputer for Meta-AI Research and...