LLMs on the Edge: The Future of On-Device Intelligence
summary
Shows a container-native, on-device pipeline—Vision LM → RAG → LLM—running rootless with Podman and GPU CDI on Jetson Orin Nano (8 GB GPU) under RHEL Edge, using Ramalama, llama.cpp, FAISS and a LiveLava 3B VLM. Claims quantized models up to 7B fit the device; required forking Ramalama to provide an arm64 CUDA image. Measures ~5.5 tokens/sec on CPU versus ~28 tokens/sec on GPU for Llama 3 3B, and demonstrates a camera-to-answer demo that runs locally with measurable first-token latency.
AI-written summary · about
chapters (9)
- 0:00Running LLMs on edge devices
- 1:05Why edge LLMs are necessary
- 2:28Edge AI constraints and challenges
- 3:52Containers for edge AI
- 5:38Podman and Rama Lama for edge AI
- 7:45GPU pass-through performance
- 9:11Live demos: voice and vision models
- 13:46Future of edge AI and agents
- 15:14Q&A
related talks
35mOn‑Device LLMs on Android Devices 2025
20mGAMING: Benchmarking Sustainable LLM Deployment on Edge Devices
47mOwn Your LLM: Running Generative and Agentic AI Locally
35mLocal LLMs on low end hardware - a practical perspective
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
22mThe Disruptive Potential of On-Device LLMs
51mJlama: A Native Java LLM inference engine
31mFrom ML to LLM on device AI in the browser