Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material that drastically changes UX design and requires reinventing methods and tools.
-
•
The Rose Bot chatbot was trained on 20 years of Rosenfeld Media’s content, making vast UX knowledge accessible.
-
•
Retrieval augmented generation enables chatbots to semantically search and embed large content sources for relevant answers.
-
•
Conversation logic layers complex steps like intent classification and safety filtering to make AI responses useful and appropriate.
-
•
Evaluation of generative AI systems demands substantial budget and time, often one-third of total resources, due to nondeterministic outputs.
-
•
Both human evaluation with experts and end users, alongside automated tools, are essential for assessing AI quality.
-
•
New research methods are needed for generative AI, as traditional UX research tools don’t scale or fit this new material.
-
•
Prompt deep dive technique involves detailed, think-aloud usability testing on single AI prompts to uncover nuanced interaction issues.
-
•
Co-creation between users and AI technology determines whether AI tools will be beneficial or harmful.
-
•
UX designers and product teams should embrace inventing new workflows, tooling, and evaluation practices for AI-driven products now.
Notable Quotes
"The moment GPT came out was the moment it changed for me as I saw my kids immediately adopt and use it at school."
"Generative AI is strangely good at some things and strangely bad at others, which makes it fascinating."
"Building a chatbot now is not scary, it’s just very different from how we've built things in the last 20 years."
"Evaluation takes about a third of project time and budget because you don’t know the input or output; it’s not deterministic."
"You have to evaluate conversation experiences with both experts and end users because usefulness and correctness don’t always align."
"The prompt deep dive technique has users spend 30 minutes on a single prompt with think-aloud to get deep insights."
"The future is co-created by users and technology, not driven purely by technological determinism."
"We need to invent new research methods and tooling because the material we are designing with is completely different now."
"It’s just as hard to build stuff that is bad for people as it is to build good things, so we might as well try to build good."
"We should feel okay to throw away old methods and keep fundamentals like deeply understanding users and systems."
Dig deeper—ask the Rosenbot:
More Videos
"Panels enable researchers to check cultural assumptions and sensitivities before launching larger studies."
Wyatt HaymanGlobal Research Panels (Videoconference)
August 8, 2020
"We investigate whether we can skin native components to respect platform conventions while identifying commonalities and differences."
PJ Buddhari Nate BaldwinMeet Spectrum, Adobe’s Design System
June 9, 2021
"Artifacts of the future help us feel and see the richness of a moment and grasp the change we hope to have."
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"Knowing your subject matter as a UX leader is just as important as knowing your craft."
Peter MerholzThe Trials and Tribulations of Directors of UX (Videoconference)
July 13, 2023
"If we continue to peak participants from homogenized groups, we risk perpetuating women’s everyday exclusions."
Dr. Jamika D. Burge Mansi GuptaAdvancing the Inclusion of Womxn in Research Practices (Videoconference)
September 15, 2022
"You have to understand how your company actually makes money."
Amy MarquezINVEST: Discussion
June 15, 2018
"The Pokémon anime was essentially a game manual, walking players through the journey with emotional stakes."
Dane DeSutter Natalie Gedeon Deborah Hendersen Cheryl PlatzBeyond the Console: The rise of the Gamer Experience and how gaming will impact UX Research across industries (Videoconference)
May 17, 2024
"Burnout really exists because we made rest a reward rather than a right."
Zariah CameronReDesigning Wellbeing for Equitable Care in the Workplace
September 23, 2024
"Ask what helps you work better—this simple question is one of the most important things for colleagues and managers."
Jessica NorrisADHD: A DesignOps Superpower
September 9, 2022
