Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive
summary
Foundry Local packages ONNX Runtime/Olive into an end-to-end on-device stack that Windows and macOS apps can use to run open models locally, claiming 10 million downloads per month for ONNX Runtime and adoption by 70,000 orgs with 1,900 models in Azure AI Foundry. Demos show model variants for CPU/CUDA/MPU, a cross‑platform summarization app using the “54 mini” model, an OCR agent that finds receipts and extracts totals, ~90 tokens/sec on one model, and >100 customers in pre‑preview.
AI-written summary · about
chapters (11)
- 0:00Introduction to Foundry Local
- 1:03Why local AI is needed
- 2:31Local AI readiness and solutions
- 5:42Foundry Local architecture
- 8:36Customer feedback on Foundry Local
- 8:56CLI demo and model exploration
- 10:39Running models and performance comparison
- 13:18Building a cross-platform summarization app
- 17:00Cross-platform app demo on Mac
- 18:06Local agent creation and demo
- 21:53Summary and next steps
related talks
39mBuild your enterprise Generative AI applications with Microsoft Olive
05mONNX Runtime
42mAI Everywhere– accelerate your development from PC to cloud
45mDeliver AI-powered experiences across cloud and edge with Windows
39mBuild your enterprise Generative AI applications with Microsoft Olive
36mApplied AI on the edge
46mAI made easier: How the ONNX Runtime and Olive toolchain will help you Q&A
30mScalable ML acceleration with ONNX Runtime