Building a self-hosted LLM ecosystem in Rust
summary
Paddler is a Rust-built, self-hosted LLM stack that actually runs as a single Rust binary with a distributed control plane and distributed load balancing across agents; each agent exposes separate management and inference addresses, uses unidirectional WebSocket traffic, and divides memory into predictable slots per agent so servers handle a known number of concurrent requests. It embeds llama.cpp via GGUF models and sidecar projection models for multimodal input, adds semantic versioning atop llama.cpp, and powers Poet, a Rust static-site generator that consumes those local LLMs.
AI-written summary · about
chapters (11)
- 0:00Padler and self-hosted LLMs
- 1:48Open-weight LLM adoption
- 3:12Admin panel and architecture
- 6:01Multi-model inputs and GGUF
- 7:46Chat templates and compatibility
- 9:09Multi-model use cases and slots
- 10:53Stability with llama.cpp
- 11:57Why Rust over PHP and Go
- 13:22Maintenance and AI dev
- 14:44Future plans and Poet
- 16:30Closing and Q&A
related talks
57mMoly - A pure Rust Explorer for Open Source LLMs
55mFast And Portable LLM Inference With WebAssembly And Rust
45mEverything You Need to Know About Running LLMs Locally
30mP2P Distributed LLM Inference Across Browsers With llama.cpp And WebRTC
40mOpen Source LLMs in the Cloud: Scalable Solutions
28mAnyscale's Unified Platform for LLM Development and Deployment
38mEmpowering AI Education with Rag: Building Domain-Specific LLM Agents for Enhanced...
39mMoxin: The All-In-One Tool to Run LLMs on Your Personal Device