Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.
Key Insights
-
•
Multimodal design involves coordinating multiple input and output modalities like visual, auditory, haptic, kinetic, and ambient to enhance user experience.
-
•
Most current product designs already incorporate multimodality, often unintentionally, especially across mobile, voice, and touch interfaces.
-
•
Cross-device design is closely linked to multimodality because users switch devices based on comfort, context, or modality limitations.
-
•
Understanding customer context through Character, Relationship, Objective, and Where helps design more inclusive and relevant experiences.
-
•
Transitions between devices and modalities—both voluntary and involuntary—are critical moments that can make or break user experience.
-
•
Network interruptions and offline states require clear, proactive communication to avoid user frustration and confusion.
-
•
Activity modeling and interruption matrices enable smart notification strategies that respect user focus and reduce disruptive interruptions.
-
•
Adaptive experiences, exemplified by USPS’s voice and web package tracking, offer seamless switching between modalities without losing task continuity.
-
•
Stakeholder buy-in for multimodal design is best achieved by aligning benefits with their goals and through shared understanding workshops.
-
•
Physical remotes remain relevant due to inclusivity (not everyone has a smartphone), lower power consumption, user habit, and purpose-driven simplicity.
Notable Quotes
"You are probably doing multimodal design already, even if you don’t realize it."
"Humans are multimodal, so the future of design has to be multimodal."
"A mode in this context is a type of communication utilizing human senses like sight, sound, movement, and touch."
"Transitions—between devices, modalities, or network states—make or break cross-device multimodal experiences."
"The USPS package tracking voice and web experience is a great example of an adaptive multimodal experience."
"If you obstruct a user’s objective at any point, that’s when you lose them."
"We need to specify how different modalities work together and hand off to create coherent experiences."
"Designing with respect to stakeholder goals and running shared understanding workshops helps align multimodal design efforts."
"Physical remotes are still relevant because phones can be stressful, die, or simply aren’t always accessible."
"Multimodal design adds a layer of orchestration on top of modality-specific design—it doesn’t replace it."
Dig deeper—ask the Rosenbot:
More Videos
"We did all these analyses for each market and then fit the group to be proportional to key hosting and traveling characteristics."
Wyatt HaymanGlobal Research Panels (Videoconference)
August 8, 2020
"Our team does not actually build the components consumed by product teams due to how Adobe's teams are structured."
PJ Buddhari Nate BaldwinMeet Spectrum, Adobe’s Design System
June 9, 2021
"We used a Wizard of Oz smoke-and-mirrors prototype to get quick buy-in from Detroit stakeholders around early childhood education access."
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"Developing trust means showing you understand what it takes to get something shipped, that you’re reliable, and that people can be vulnerable with you."
Peter MerholzThe Trials and Tribulations of Directors of UX (Videoconference)
July 13, 2023
"Feeling heard and seen by research practitioners is one of the best compliments a participant can give."
Dr. Jamika D. Burge Mansi GuptaAdvancing the Inclusion of Womxn in Research Practices (Videoconference)
September 15, 2022
"Investment is attention. If the business funds you, they believe in what you’re doing."
Amy MarquezINVEST: Discussion
June 15, 2018
"Gaming is about generating positivity in ways software like Azure can’t."
Dane DeSutter Natalie Gedeon Deborah Hendersen Cheryl PlatzBeyond the Console: The rise of the Gamer Experience and how gaming will impact UX Research across industries (Videoconference)
May 17, 2024
"Play invites curiosity, and curiosity allows us to imagine and create new worlds."
Zariah CameronReDesigning Wellbeing for Equitable Care in the Workplace
September 23, 2024
"Ask what helps you work better—this simple question is one of the most important things for colleagues and managers."
Jessica NorrisADHD: A DesignOps Superpower
September 9, 2022
