Applied AI and Accessibility to Play Games in New Ways
summary
A platform built in .NET on Azure turns voice, face expressions, head/eye gestures and phone taps into keyboard/mouse/gamepad inputs and ships as Project Enabled Play / seable; early prototypes use Arduino USB HID, SignalR and Alexa skills and a Snapchat lens hack, later replaced by on‑device ONNX models and a Llama 2 + Intel control generator. Predictive ASR runs on edge at ~30 ms (vs 1.5 s cloud roundtrips), cutting cloud bills that hit ~$30k/month and enabling users like a seven‑year‑old with AFM to score 10 in a game, so the approach works. ONNX models
AI-written summary · about
chapters (9)
- 0:00AI and gaming accessibility
- 10:11AI to play Dark Souls
- 16:12Voice-controlled gaming with Arduino
- 22:08Prototype to hackathon project
- 29:08Prototype to real-world solutions
- 36:47Distributed controls and authentication
- 41:06Adaptive controls with ML and LLMs
- 44:11Live face recognition with ONNX
- 55:21AI tools and takeaways
related talks
55mMaking Every Game and App More Accessible with .NET and AI
36mMaking Technology More Accessible with .NET and AI
42mMaking Every Game More Accessible with .NET and AI
47mAccessibility in the era of generative AI
40mMicrosoft AI for Inclusion and Innovation
46mBuilding accessible experiences with AI
17mHow Cephable Applies On-Device AI to Accessibility
44mThe age of applied AI