Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material that drastically changes UX design and requires reinventing methods and tools.
-
•
The Rose Bot chatbot was trained on 20 years of Rosenfeld Media’s content, making vast UX knowledge accessible.
-
•
Retrieval augmented generation enables chatbots to semantically search and embed large content sources for relevant answers.
-
•
Conversation logic layers complex steps like intent classification and safety filtering to make AI responses useful and appropriate.
-
•
Evaluation of generative AI systems demands substantial budget and time, often one-third of total resources, due to nondeterministic outputs.
-
•
Both human evaluation with experts and end users, alongside automated tools, are essential for assessing AI quality.
-
•
New research methods are needed for generative AI, as traditional UX research tools don’t scale or fit this new material.
-
•
Prompt deep dive technique involves detailed, think-aloud usability testing on single AI prompts to uncover nuanced interaction issues.
-
•
Co-creation between users and AI technology determines whether AI tools will be beneficial or harmful.
-
•
UX designers and product teams should embrace inventing new workflows, tooling, and evaluation practices for AI-driven products now.
Notable Quotes
"The moment GPT came out was the moment it changed for me as I saw my kids immediately adopt and use it at school."
"Generative AI is strangely good at some things and strangely bad at others, which makes it fascinating."
"Building a chatbot now is not scary, it’s just very different from how we've built things in the last 20 years."
"Evaluation takes about a third of project time and budget because you don’t know the input or output; it’s not deterministic."
"You have to evaluate conversation experiences with both experts and end users because usefulness and correctness don’t always align."
"The prompt deep dive technique has users spend 30 minutes on a single prompt with think-aloud to get deep insights."
"The future is co-created by users and technology, not driven purely by technological determinism."
"We need to invent new research methods and tooling because the material we are designing with is completely different now."
"It’s just as hard to build stuff that is bad for people as it is to build good things, so we might as well try to build good."
"We should feel okay to throw away old methods and keep fundamentals like deeply understanding users and systems."
Or choose a question:
More Videos
"The health of our planet directly influences our health and survival."
Alex Hurworth Bonnie John Fahd Arshad Antoine MarinDesigning a Contact Tracing App for Universal Access
October 23, 2020
"It’s really okay if this new position on your team is short-term only; we can provide the initial opportunity to get started on their career paths."
Laine Riley Prokay Lisa GordonCarving a Path for Early Career DesignOps Practitioners
September 9, 2022
"TripAdvisor is really nine major business units each thinking like sub companies instead of one end-to-end user experience."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"Design researchers know customers better than almost anyone else in the organization, yet they are rarely invited into strategy processes."
Nathan ShedroffDouble Your Mileage: Use Your Research Strategically
March 31, 2020
"Apple premiered mobile accessibility in a very exciting way with the iPhone 3GS and VoiceOver announcement at WWDC."
Sam ProulxMobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
November 17, 2022
"Foundational research answers big, nebulous questions; rapid research focuses on specific usability questions."
Feleesha SterlingBuilding a Rapid Research Program (Videoconference)
May 18, 2023
"We are at an inflection point where what got us here won't get us to a thriving future."
Neil BarrieWidening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
March 25, 2024
"Incremental improvements and disruptive innovation require very different methods and measures."
John DevanneyThe Design Management Office
November 6, 2017
"You need to understand stakeholders’ fears, motivations, and incentives to change hearts and minds."
Katy MogalBut Do Your Insights Scale?
March 12, 2021