Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Case Study] Don't botch the bot: Designing interactions for AI
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Case Study] Don't botch the bot: Designing interactions for AI
Speakers: Savannah Carlin
Link:

Summary

It seems like every company is adding a conversational AI chatbot to their website lately, but how do you actually go about making these experiences valuable and intuitive? Savannah Carlin will present a case study on a conversational AI chatbot—Marqeta Docs AI—that she designed for a developer documentation site in the fintech industry. She will share her insights, mistakes, and perspectives on how to use AI in a meaningful, seamless way, especially for companies like Marqeta that operate in highly regulated industries with strict compliance standards. The talk will use specific examples and visuals to show what makes conversational AI interactions uniquely challenging and the design patterns that can address those challenges. These include managing user expectations, handling errors or misunderstandings within the conversation, and ensuring that users can quickly judge the quality of a bot’s response. You’ll gain a deeper understanding of the intricacies involved in designing interactions for AI, along with practical advice you can apply in your own design processes. Take-aways What to consider before you add AI to your product to ensure it will be valuable, usable, and safe for its intended workflows The interactions that are unique to conversational AI experiences and the design patterns that work for them Common challenges in designing conversational AI experiences and how to overcome them

Key Insights

  • Clearly define the primary use case before building a generative AI tool to ensure its relevance and usefulness.

  • High-quality, thoroughly vetted training data is foundational for trustworthy AI outputs, especially in regulated domains like FinTech.

  • Properly framing the initial state of the chatbot guides users to ask relevant questions and reduces irrelevant or out-of-scope interactions.

  • Loading indicators should be subtle; show loading dots before text appears and let the text rendering itself serve as feedback.

  • Supporting easy scrolling and navigation between prompt input and AI output helps users refine their queries effectively.

  • Error handling in AI chatbots shifts from fixing technical failures to coaching users on better prompt writing.

  • Transparency about AI accuracy, including disclaimers and source citations, is crucial to build and maintain user trust.

  • Accessibility must be integrated from the start, including keyboard navigation and screen reader support, especially given long conversational outputs.

  • User feedback mechanisms like thumbs up/down and source link interactions produce valuable data to iteratively improve chatbot performance.

  • Narrowing the chatbot’s scope at launch reduces risks and false outputs, with future plans to expand question domains carefully.

Notable Quotes

"You need to be very clear on the primary use case and what this tool will help someone do."

"If you have any doubts about the quality of the training data, do not proceed. Do not pass, go."

"When someone arrives at this tool, how do they know what to use it for? Framing the interaction is key."

"Loading dots should only appear before any letters show up; the text appearing itself acts as a loading indicator."

"People often forget what they typed and want to scroll back to their prompt to refine it."

"Error states are less about technical errors and more about helping people write prompts effectively."

"Being transparent about the tool’s accuracy and limitations is especially crucial in FinTech due to regulations."

"Every single output should have at least three source links to allow users to verify information."

"We made sure every person could navigate the chatbot using a keyboard alone to support accessibility."

"People were asking questions they might never have emailed us about, decreasing friction to learn more."

Amanda Kaleta-Kott
The Joys and Dilemmas of Conducting UX Research with Older Adults
2022 • Advancing Research 2022
Gold
Etienne Fang
The Power of Care: From Human-Centered Research to Humanity-Centered Leadership
2021 • Advancing Research 2021
Gold
Robin Beers
Research as a Catalyst for Organizational Transformation
2021 • Advancing Research 2021
Gold
Maish Nichani
Sparking a Service Excellence Mindset at a Government Agency
2021 • Civic Design 2021
Gold
Sarah Barrett
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
2025 • Rosenfeld Community
Holly Cole
Understanding Experiences: When you have to do more than work
2018 • DesignOps Summit 2018
Gold
Jose Coronado
People First - Design at JP Morgan
2021 • Design at Scale 2021
Gold
Jemma Ahmed
Theme 2 Intro
2024 • Advancing Research 2024
Gold
Laura Weiss
There is No Playbook: Leader as Coach During Challenging Times (Videoconference)
2024 • Rosenfeld Community
Emilia Åström
Unlock Your Team’s Intelligence with Collaboration Design
2022 • Design at Scale 2022
Gold
John Calhoun
Meters, Miles, and Madness: New Frameworks to Measure the (Elusive) Value of DesignOps
2024 • DesignOps 2024
Gold
Llewyn Paine
[Demo] Deploying AI doppelgangers to de-identify user research recordings
2024 • Designing with AI 2024
Gold
Sam Proulx
Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
2022 • DesignOps Summit 2022
Gold
Emily Lessard
RFPs Without Tears: Writing Inclusive RFPS that Don't Scare Away Talent
2021 • Civic Design 2021
Gold
Dane DeSutter
What co-speech gestures reveal about users’ thinking during interviews (Videoconference)
2023 • Rosenfeld Community
Cheryl Platz
Merging Improv with Design (Videoconference)
2019 • Enterprise Community

More Videos

Daniela Magaña Flores

"The Product React uses reaction time to capture the user’s gut feeling that otherwise remains hidden."

Daniela Magaña Flores Ariane Rahn Jeff Ephraim Bander

Ahead of Competition: Learn What UX Benchmarking Can Do for Your Business Today

March 10, 2022

Savina Hawkins

"The secret sauce behind GPT’s abilities is the transformer architecture, the latest in kitchen technology that allows the chef to work really fast and efficiently."

Savina Hawkins

Harnessing AI in UXR: Practical Strategies for Positive Impact

March 26, 2024

Alex Hurworth

"We are witnessing the most rapid loss of biodiversity in human history."

Alex Hurworth Bonnie John Fahd Arshad Antoine Marin

Designing a Contact Tracing App for Universal Access

October 23, 2020

Matt Duignan

"Durability of insights varies; some human factors last a long time while cultural elements like hamburger menus shift over time."

Matt Duignan

HITS, Microsoft's internal human insight system: From research library to living body of knowledge (Videoconference)

July 16, 2019

Craig Villamor

"The only form of ethical persuasion that exists is when the goals of the persuader are aligned with the goals of the persuading - Tristan Harris."

Craig Villamor

Design Systems for Ethical Design (Videoconference)

January 26, 2023

Ovetta Sampson

"If you don't think that you can go to battle day in and day out, UX might not be for you because that's what it is."

Ovetta Sampson

Turning UX Passion into Real Product Influence

June 7, 2023

Rima Campbell

"To advance design maturity, you need a tight business partnership between product management and design ops leaders."

Rima Campbell Amrit S Bhachu

Increase Productivity and Drive Business Impact

September 24, 2024

Amy Gawronski Zuccaro

"We had to negotiate getting Figma behind our single sign-on to ensure security, and it took multiple calls."

Amy Gawronski Zuccaro

Advice for DesignOps Employee #1

September 29, 2021

Victor Udoewa

"The skills of research sit equal to and alongside all the talents and gifts of community members."

Victor Udoewa

Beyond Methods and Diversity: The Roots of Inclusion

March 26, 2024