Surviving the Low Memory Killer with Local AI Runtimes
summary
Production architecture for surviving LMKD that intercepts ComponentCallbacks2 signals to halt allocations, offloads tensors out of the Java heap into shared memory (demo moves 48 MB from heap to a shared tensor), and memory maps large weights with mmap so demand paging lets a 12‑engine pool run where it otherwise fails. Uses direct/native buffers and ByteBuffer.allocateDirect, routes compute to NNAPI / Light RT NPUs, queries getThermalHeadroom, and adds WAL checkpoints for transactional resume.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:57Local AI runtimes necessity
- 2:01Auto-regressive generation & KV cache
- 3:46Android memory & LMKD
- 5:55LMKD app termination
- 7:23ComponentCallbacks2 for memory
- 10:38Java heap & garbage collection
- 12:01Shared memory offloading
- 14:07mmap virtual memory
- 17:19Hardware for AI models
- 19:47Thermal throttling & APIs
- 20:49WAL logging resiliency
- 21:52Memory optimization recap
- 23:17Q&A
related talks
26mMemory-Efficient LLM Inference on Edge Devices With NNTrainer
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
26mLMCache: Lower LLM Performance Costs in the Enterprise
17mCLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
49mTrash Talk
42mStop Treating LLMs Like REST APIs
15mResource Multiplexing in Tuning and Serving Large Language Models
25mWhat does it take to build long-term state for AI agents?