Interactive Planning-based Cognitive Assistance on the Edge
summary
System tracks actions by classifying the top object and uses user confirmation when sequences are ambiguous, then runs a PDDL planner only on deviations while a generated state machine serves the common path. Prototype runs on Android and HoloLens 2 with an edge server for offload, uses a vision classifier at 20 ms per image, and a planner implemented in PDDL; experiments show planning latency is small enough for real-time guidance and the sandwich assembly demo recovers from user mistakes.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:28What is cognitive assistance?
- 1:07How to build a cognitive assistant?
- 3:58How about a planner?
- 4:58A planning problem and goal state
- 6:19Ambiguity Resolving
- 7:35Dynamic State Tracking
- 8:40Runtime of the planner and classifie
- 10:10Future Work
related talks
47mPlan Before You Build: Deterministic Planning Patterns for AI Agents
03mAutonomy and Human-AI Interaction
22mAI-Powered Personalized Learning
12mScaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initi...
13mGuided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
13mInteractive Learning and User‑Centered AI: A Retrospective
08mReal AI Agents Need Planning, Not Just Prompting
13mProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Moda...