Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
summary
Geno is a JavaScript/Electron IDE that trains a Rasa-backed language model and uses the WebSpeech API to add voice commands to existing web apps by mapping intents to existing functions or replayed UI actions. Developers with little NLP experience add multimodal parameters by extracting HTML attributes; in an evaluation 6 of 8 participants complete four tasks (calendar and music player) under an hour. Current limits: one intent → one action, single-parameter multimodal context, and only single-page desktop apps supported.
AI-written summary · about
related talks
34mUsing Appsmith for building GenerativeAI apps
30mLook ma, no hands! Multimodal AI-Agents in the browser
33mThe Future of Developer Experience with GenAI: Driving Engineering Excellence
25mMulti-Talking (not a typo!) with Genaver
19m"Bringing Voice First Experiences to the Web"
38mCreate the next generation of user experiences with text generation
59mThe Evolution of the Player Interface.
19mThe Physics of Requirement Sciences Through Application [...]