PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and ...
summary
PUMICE is a multi-modal agent that learns new concepts and conditionals by combining natural language with programming-by-demonstration, using PUMICE built on the SUGILITE demonstration engine and SUGILITE for GUI manipulation. It stores procedures, value queries, and Boolean concepts in a persistent knowledge base, generalizes by parameterizing GUI structures and type-based highlighting, and completes four scripted teaching tasks in a lab study with 10 users who all finish every task in about 5–7 minutes, showing practical usability.
AI-written summary · about
chapters (9)
- 0:00Introduction to PUMICE
- 1:14Problem with intelligent agents
- 3:46Design goals and approach
- 6:42Demo of PUMICE system
- 9:15How PUMICE learns concepts
- 10:40Knowledge types and generalization
- 13:11User study results
- 14:16Future work directions
- 14:36Conclusion and Q&A
related talks
28mTalking to Objects in Natural Language: Toward Semantic Tools for Exploratory(…)
39mIn situ model-based learning in PAMELA
24mThe truth about AI development
12mScaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initi...
41m"Natural Interfaces" and "Natural Language Programming"
20mUseful General Intelligence
26mDSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
13mInteractive Learning and User‑Centered AI: A Retrospective