From Pixels To Agents: Optimizing On-Device Performance of Confidential Computing in AI Evolution
summary
Experiments on ISLET/isolate show confidential on-device AI adds measurable costs: ~10% inference overhead from frequent context switches, and model loading up to 90% slower due to repeated host-CC handoffs. A pre-mapping reduces loading by ~80% by allocating and loading parameters at init, making CC practical for fixed models. Agent designs trade isolation versus latency: single CC VM keeps zero comm cost but no module isolation, multiple CC VMs protect memory at added messaging cost that use WebSocket channels can cut.
AI-written summary · about
chapters (9)
- 0:00On-device confidential AI intro
- 4:17Confidential computing for AI
- 7:10Overheads in confidential envs
- 9:20Inference overheads in CC
- 15:07Faster model loading with premapping
- 17:39Optimizing model size
- 20:52Environments communication challenges
- 22:00Agentic AI memory
- 26:20Key takeaways and future
related talks
42mSponsored Session: Confidential Computing - New Capabilities for New Workloads
14mFaster AI model training with confidential computing
29mCloud and Edge: Price, Performance and Privacy Considerations in IOT
36mApplied AI on the edge
41mEscaping the Cloud: High-Performance AI in your Browser
47mAI and Edge Computing: Efficient, Lean, Dependency‑Free Language Processing
17mFrom Agents to Action: Scaling Agentic AI with Ray on Intel Xeon
19mReal-Time AI: Low Latency Solutions for Interactive Applications