Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Case Study] Don't botch the bot: Designing interactions for AI
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Case Study] Don't botch the bot: Designing interactions for AI
Speakers: Savannah Carlin
Link:

Summary

It seems like every company is adding a conversational AI chatbot to their website lately, but how do you actually go about making these experiences valuable and intuitive? Savannah Carlin will present a case study on a conversational AI chatbot—Marqeta Docs AI—that she designed for a developer documentation site in the fintech industry. She will share her insights, mistakes, and perspectives on how to use AI in a meaningful, seamless way, especially for companies like Marqeta that operate in highly regulated industries with strict compliance standards. The talk will use specific examples and visuals to show what makes conversational AI interactions uniquely challenging and the design patterns that can address those challenges. These include managing user expectations, handling errors or misunderstandings within the conversation, and ensuring that users can quickly judge the quality of a bot’s response. You’ll gain a deeper understanding of the intricacies involved in designing interactions for AI, along with practical advice you can apply in your own design processes. Take-aways What to consider before you add AI to your product to ensure it will be valuable, usable, and safe for its intended workflows The interactions that are unique to conversational AI experiences and the design patterns that work for them Common challenges in designing conversational AI experiences and how to overcome them

Key Insights

  • Clearly define the primary use case before building a generative AI tool to ensure its relevance and usefulness.

  • High-quality, thoroughly vetted training data is foundational for trustworthy AI outputs, especially in regulated domains like FinTech.

  • Properly framing the initial state of the chatbot guides users to ask relevant questions and reduces irrelevant or out-of-scope interactions.

  • Loading indicators should be subtle; show loading dots before text appears and let the text rendering itself serve as feedback.

  • Supporting easy scrolling and navigation between prompt input and AI output helps users refine their queries effectively.

  • Error handling in AI chatbots shifts from fixing technical failures to coaching users on better prompt writing.

  • Transparency about AI accuracy, including disclaimers and source citations, is crucial to build and maintain user trust.

  • Accessibility must be integrated from the start, including keyboard navigation and screen reader support, especially given long conversational outputs.

  • User feedback mechanisms like thumbs up/down and source link interactions produce valuable data to iteratively improve chatbot performance.

  • Narrowing the chatbot’s scope at launch reduces risks and false outputs, with future plans to expand question domains carefully.

Notable Quotes

"You need to be very clear on the primary use case and what this tool will help someone do."

"If you have any doubts about the quality of the training data, do not proceed. Do not pass, go."

"When someone arrives at this tool, how do they know what to use it for? Framing the interaction is key."

"Loading dots should only appear before any letters show up; the text appearing itself acts as a loading indicator."

"People often forget what they typed and want to scroll back to their prompt to refine it."

"Error states are less about technical errors and more about helping people write prompts effectively."

"Being transparent about the tool’s accuracy and limitations is especially crucial in FinTech due to regulations."

"Every single output should have at least three source links to allow users to verify information."

"We made sure every person could navigate the chatbot using a keyboard alone to support accessibility."

"People were asking questions they might never have emailed us about, decreasing friction to learn more."

Monty Hammontree
The Future of UX Research (Videoconference)
2020 • Advancing Research Community
Peter Merholz
Customer-Centered Design Organizations
2017 • Enterprise Experience 2017
Gold
Jessamyn Edwards
Surviving Your UX Career in Enterprise Design (Videoconference)
2021 • Enterprise Community
Dantley Davis
Leadership & Diversity—A Fireside Chat with Dantley Davis (Videoconference)
2020 • Enterprise Community
Simon Wardley
Maps and Topographical Intelligence (Videoconference)
2019 • Enterprise Community
Sha Hwang
The Lost Year
2021 • Design at Scale 2021
Gold
Alexis Lucio
Scaling Accessibility Through Design Systems
2022 • Design at Scale 2022
Gold
Kate Kalcevich
Designing inclusively with AI
2024 • Designing with AI 2024
Gold
Louis Rosenfeld
Coffee with Lou: Should You Write a (UX) Book? (Videoconference)
2024 • Rosenfeld Community
Rittika Basu
Age and Interfaces: Equipping Older Adults with Technological Tools (Videoconference)
2023 • Advancing Research Community
Alexandra Schmidt
Why Ethics Can't Save Tech
2022 • Civic Design 2022
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • DesignOps Summit 2021
Gold
Kit Unger
Theme 1 Intro
2022 • Design at Scale 2022
Gold
Dr. Jamika D. Burge
How UX researchers can partner with (and not be replaced by) AI [Advancing Research Community Workshop Series](Videoconference)
2023 • Advancing Research Community
Shahrzad Samadzadeh
What Is My Value? Two Takes and Some Mistakes
2024 • Enterprise Experience 2020
Gold
Stretching the Definition of DesignOps with Product Development
2018 • DesignOps Summit 2018
Gold

More Videos

Monty Hammontree

"We changed the language to revolve around assumptions, hypotheses, experiments, and making sense of data."

Monty Hammontree

The Future of UX Research (Videoconference)

December 3, 2020

Liz Ebengo

"Will you become an advocate for the world’s most vulnerable voices muffled in complex systems?"

Liz Ebengo

The Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward

December 4, 2024

Billy Carlson

"Staying in low fidelity throughout early concept development lets you focus strictly on the problem and solution space."

Billy Carlson

Tips to Utilize Wireframes to Tell an Effective Product Story

June 6, 2023

Saara Kamppari-Miller

"Nothing about us without us means bringing people in throughout the research and design processes."

Saara Kamppari-Miller

Inclusive Design is DesignOps

September 29, 2021

Ian Swinson

"People care so much about titles, but honestly, I’d prefer the star-bellied sneak on the business card."

Ian Swinson

Designing and Driving UX Careers

June 8, 2016

Jonathon Colman

"Working across multiple teams all the time is normal for content designers, but that doesn’t mean it’s the best or the only way."

Jonathon Colman

How to Maximize the Impact of Content Design

January 8, 2024

Maria Skaaden

"Google Design Sprint is the best inception strategy I know for design thinking because you learn without thinking about learning."

Maria Skaaden

Continuous Design: One eye on the horizon and the other on the next wave

November 8, 2018

Bryce Benton

"I learned from Molly that the lunatic asylum is now called the Austin State Hospital, and I've encoded that in a Python script."

Bryce Benton

[Demo] AI-powered UX enhancement: Aligning GitHub documentation with USWDS at Austin Public Library

June 4, 2024

Marc Fonteijn

"This report is really about finding clues to make smarter career decisions, not necessarily about giving you definitive answers."

Marc Fonteijn

First Insights from the 2025 Service Design Salary(+) Report

December 4, 2024