Signals-Based Testing Of Microsoft Office With An AI-Driven Robot Army
summary
Builds an AI-driven robot army that interacts with Office by emitting labeled state-action-state traces, uploading them to a server that trains models to return sequences that hit rewards (UI events like clicking Bold, opening Print Merge). Modifies Office to mark sessions as under test and adds automation IDs and seeded files to boost reach, achieving roughly 70% coverage after weeks on thousands of machines. Finds thousands of crashes and a mid-six-digit universe of distinct crash IDs, forcing call-stack classifiers, replayability fixes, and heavy prioritization.
AI-written summary · about
chapters (13)
- 0:00Signals-based testing intro
- 1:28Office testing with signals
- 3:57AI testing evolution at Microsoft
- 5:01AI robot army learns workflows
- 8:40Reinforcement learning training
- 10:06Monitoring AI test problems
- 12:31Scaling AI testing challenges
- 15:43Crashes found by AI testing
- 17:30Prioritizing AI failures
- 21:43Predictive crash modeling
- 23:30AI replacing manual testing myths
- 27:06AI testing challenges
- 30:38Q&A
related talks
16mDon't just talk to AI, do more with AI: how to improve productivity with AI agents
40mDavid Fogl: Leveraging Generative AI for Enhanced Software Testing
54mAI for Software Testing: The Ultimate Journey
22mMaking AI Accountable: Azure AI Foundry
01h 02mKeynote - Building for the Future: Microsoft 365, Agents, & AI
15mPractical tactics to build reliable AI apps
40mFrom Bugs to Brilliance: Leveraging AI in Modern Software Testing
10mAI Co-Pilot Demo