ERROR IN PROGRESS
CONTRIBUTOR
Ka Young Lee
MENTOR
Melanie Bossart
CATEGORY
MA Interaction Design Thesis Project | Multimodal Ai Assistant Application
FULL DOCUMENTATION >>
HOW
EP's research approach combined surveys, interviews, and hands-on workshops to understand how users experience and recover from voice assistant errors.
1:1 interviews explored how users interacted with advanced multimodal assistants like Hume AI and ChatGPT-4o. While ChatGPT-4o offered some adaptability, both systems struggled with bilingual input and failed to clarify intent effectively.
A workshop using a collaborative origami task highlighted the limits of voice-only instructions. Participants relied on switching modes or adding visual cues—reinforcing the need for multimodal support, progressive feedback, and clearer system responses.
RESULT
EP culminates in a multimodal voice assistant prototype that redefines how users navigate and recover from interaction errors. Instead of relying solely on voice, the system introduces visual and tactile channels to make misunderstandings visible—and correctable.
By implementing selective speech-to-text editing and progressive visual feedback, EP shifts the dynamic from passive dictation to active collaboration. Users can now identify where the assistant misunderstood them and make targeted corrections without starting over, reducing frustration and increasing trust.
The result is a voice assistant that doesn’t just react—it responds with clarity, context, and adaptability. EP showcases how designing for error isn’t a limitation, but an opportunity to strengthen the human-machine relationship.
_gif.gif)
