What I Wish I Knew Before Building On-Device AI Features
summary
Built an on‑device feedback classifier and an attempted chat clone using Apple Foundation Models, discovering the 4096 token limit is input+instructions+tool calls, not just user text. Sampling controls matter—greedy, top‑K, top‑P with a seed change determinism—so choose per use case. Estimate tokens by divide text length by four and trim with summarization or a sliding window. Use persistent sessions for multilingual guardrails, pre‑warm on user typing to save battery and avoid models becoming unavailable under thermal or low‑power conditions.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:26What this talk covers and disclaimers
- 5:13Feedback classification with prompts
- 9:53Non-deterministic LLM outputs
- 10:58Sampling controls for determinism
- 13:54Sampling method selection
- 16:28Token limits and context window
- 18:57Token estimation for UX
- 20:43LLM safety practices
- 23:33Disclaimers for AI features
- 24:16Multilingual inputs handling
- 27:24Pre-warming models
- 28:49Model unavailability risks
- 29:32Recap of key takeaways and best practices
- 31:16Knowledge sharing
related talks
21mBeyond the Chatbox: What Changes When AI Runs Where You Are
59mKeynote: AI without the BS, for humans
56mIron Man or Ultron: Is AI here to help us or hurt us?
48mMastering the AI Shift: Adaptive Strategies for Software Evolution
38mCreate the next generation of user experiences with text generation
21mFrom 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents
44mAI Foundations: Mitigate Risks and Boost SOC Efficiency
24mBring the Power of AI to Your Application