SketchGPT: A Sketch-based Multimodal Interface for Application-Agnostic LLM Interaction
summary
SketchGPT is a sketch+speech interface that wires sketches, audio, and on-screen context into a multi-agent framework that segments and aligns modalities, uses Chain-of-Thought style reasoning, and issues toolkit-aware, executable actions with human-in-loop confirmation. The team builds a working prototype that generates images, edits documents, and offers a selectable manipulation list. In user studies SketchGPT yields higher input efficiency, lower mental and temporal demand, and average usability ratings above 5.5/7; main failures are spatial errors.
AI-written summary · about
chapters (7)
- 0:00Introduction to SketchGPT
- 1:17Formative study findings
- 3:51System implementation details
- 4:56Real-world use cases demo
- 6:46User study results
- 8:30Future work directions
- 9:56Q&A session
related talks
05mChatGPT is poorly designed. So I fixed it
10mSGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents
07mSketchar: Supporting Character Design and Illustration Prototyping Using Generative AI
14mA Framework for Motivating Sketching Practice with Sketch-based Gameplay
45mHow to use GPT in your projects: A Case Study
39mPragmatic Sketching
49mPragmatic Sketching
02h 00mChatGPT & Prompt Engineering