Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing

Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Alëna Iouguina
Designing Systems at Scale
2018 • DesignOps Summit 2018
Gold
Changying (Z) Zheng
Practical DesignOps: From Ideas to Tools That Teams Actually Use
2025 • Rosenfeld Community
Matt LeMay
You Don’t “Get” Anyone to Do Anything
2022 • Design in Product 2022
Gold
Cornelius Rachieru
Handling Complexity: Framing a Scale of Design
2021 • Design at Scale 2021
Gold
Bria Alexander
Theme Two Intro
2023 • DesignOps Summit 2023
Gold
Mohammad Hossein Jarrahi
Contextuality problem: Exploring the Benefits of Qualitative and Quantitative Research
2023 • QuantQual Interest Group
Andrew Michael
Building a Product Insights Team
2022 • Advancing Research 2022
Gold
Uday Gajendar
Day 2 Welcome
2024 • Designing with AI 2024
Gold
George Abraham
Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)
2021 • DesignOps Summit 2021
Gold
Kim Fellman Cohen
Measuring the Designer Experience
2019 • DesignOps Summit 2019
Gold
Taylor Jennings
Research Debate Club
2026 • Advancing Research 2026
Gold
Sarit Geertjes
People, not Petri Dishes: Stories from a Research Recruiter
2019 • DesignOps Community
Onur Kocan
Understanding the Strategy for Civic Design in a Complex City: Istanbul
2022 • Civic Design 2022
Gold
Clara Kliman-Silver
UX Futures: The Role of Artificial Intelligence in Design
2023 • Enterprise UX 2023
Gold
Marjorie Stainback
Transforming Strategic Research Capacity through Democratization
2019 • DesignOps Summit 2019
Gold
Frances Yllana
DesignOps Exposed: What do our peers really think of us?
2025 • DesignOps Summit 2025
Gold

More Videos

Taylor Jennings

"We had to convince our director and executives by showing how the repository added real value for the product team."

Taylor Jennings Joe Nelson Alex Knoll

Repository Retrospective: Learnings from Introducing a Central Place for UX Research

March 9, 2022

Nicole Aleong

"We have a role to play in democratizing the future and amplifying more visions of what that future can hold."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Jeff Gothelf

"The innovation studio bakes in strategic vision, leadership, funding, and ownership."

Jeff Gothelf

Innovation Studios: the Engines of Enterprise Experimentation

May 14, 2015

Saara Kamppari-Miller

"Leadership needs to be just as behind accessibility as the designers and professionals doing the work."

Saara Kamppari-Miller

DesignOps for Inclusive Design and Accessibility

May 26, 2022

Tess Dixon

"Design ops folks meet the habit of catering so much to everybody’s increasingly trivial whims that it takes us away from the real meaty ops work."

Tess Dixon

C'mon Get Happy

September 29, 2021

Liam Thurston

"If your team sees a clear path of mastery in your organization, they’re much more likely to stay."

Liam Thurston

Why Your Design Team Is Quitting, And How To Fix It

June 10, 2022

Ian Swinson

"It’s huge to have a framework so managers and employees speak the same language around skills and growth."

Ian Swinson

Designing and Driving UX Careers

June 8, 2016

Leisa Reichelt

"At Atlassian, many confident non-researchers were already doing lots of research, which was an opportunity and a challenge."

Leisa Reichelt

Opening Keynote: Operating in Context

November 7, 2018

Rachael Greene

"Getting to know someone as a person first—the vibe check—is the first step to successful UX research collaboration."

Rachael Greene Alison Davis

Building a Design Ops Practice that Really Works (Most of the Time)

October 2, 2025