Safe Evaluation and Rollout of AI Models
summary
Azure Copilot for the Azure portal uses foundation models with retrieval augmented generation (no fine‑tuning) and ships through canary dogfooding and 1% experiments. Feedback is thumbs up/down plus NSAT/NPS; conversation sharing defaults off for privacy, enabled by default for corporate logins to get debug data. LLMs generate 10k test prompts and auto‑evaluate responses, model swaps feel like rearchitectures, and about 100 handlers route domain answers; the system is in preview heading toward general availability.
AI-written summary · about
chapters (9)
- 0:00AI rollout challenges
- 3:23Non-experts building AI
- 8:01AI reliability and testing
- 12:57User feedback for reliability
- 18:35Measuring AI satisfaction
- 24:58Experimenting in production
- 30:13Testing AI prompt changes
- 33:45Model upgrades rearchitectures
- 34:28Observability for AI handlers
related talks
42mBuilding and using AI models responsibly Q&A
45mThe era of the AI Copilot
48mDeploy large language models responsibly with Azure AI
40mAI Studio - Creating and scaling your custom copilots
48mUsing Azure AI Foundry to manage all your Large Language Models
20mAgentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK
22mMaking AI Accountable: Azure AI Foundry
48mGetting started with generative AI using Azure OpenAI Service