Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material that drastically changes UX design and requires reinventing methods and tools.
-
•
The Rose Bot chatbot was trained on 20 years of Rosenfeld Media’s content, making vast UX knowledge accessible.
-
•
Retrieval augmented generation enables chatbots to semantically search and embed large content sources for relevant answers.
-
•
Conversation logic layers complex steps like intent classification and safety filtering to make AI responses useful and appropriate.
-
•
Evaluation of generative AI systems demands substantial budget and time, often one-third of total resources, due to nondeterministic outputs.
-
•
Both human evaluation with experts and end users, alongside automated tools, are essential for assessing AI quality.
-
•
New research methods are needed for generative AI, as traditional UX research tools don’t scale or fit this new material.
-
•
Prompt deep dive technique involves detailed, think-aloud usability testing on single AI prompts to uncover nuanced interaction issues.
-
•
Co-creation between users and AI technology determines whether AI tools will be beneficial or harmful.
-
•
UX designers and product teams should embrace inventing new workflows, tooling, and evaluation practices for AI-driven products now.
Notable Quotes
"The moment GPT came out was the moment it changed for me as I saw my kids immediately adopt and use it at school."
"Generative AI is strangely good at some things and strangely bad at others, which makes it fascinating."
"Building a chatbot now is not scary, it’s just very different from how we've built things in the last 20 years."
"Evaluation takes about a third of project time and budget because you don’t know the input or output; it’s not deterministic."
"You have to evaluate conversation experiences with both experts and end users because usefulness and correctness don’t always align."
"The prompt deep dive technique has users spend 30 minutes on a single prompt with think-aloud to get deep insights."
"The future is co-created by users and technology, not driven purely by technological determinism."
"We need to invent new research methods and tooling because the material we are designing with is completely different now."
"It’s just as hard to build stuff that is bad for people as it is to build good things, so we might as well try to build good."
"We should feel okay to throw away old methods and keep fundamentals like deeply understanding users and systems."
Or choose a question:
More Videos
"We had no budget, no backing, but we started looking at competitors and created something ourselves."
Jennifer KanyamibwaCreating the Blueprint: Growing and Building Design Teams
November 8, 2018
"We want to create clarity because lack of clarity is why we struggle with research repositories."
Brigette Metzler Dana ChrisfieldResearch Repositories: A global project by the ResearchOps Community (Videoconference)
August 27, 2020
"Sometimes you just have to do what you can and come back to it later when analyzing culture versus delivery demands."
Carl TurnerYou Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
March 28, 2023
"Gina was closing the dots and shifting perspective from a designer closer to the business."
John Mortimer Milan Guenther Lucy Ellis Patrick QuattlebaumPanel Discussion
December 3, 2024
"An engaged designer is a retained designer."
Dante GuintuHow to Crush the Talent Crunch
September 8, 2022
"Fourth order design is about understanding the system structure and how we intervene to create new pathways of experience."
Richard BuchananCreativity and Principles in the Flourishing Enterprise
June 15, 2018
"Saul Metz is going to give advice on how to hire other outsiders like her."
Dan WillisTheme 3: Intro
January 8, 2024
"Failure is only the beginning."
Dan WardFailure Friday #1 with Dan Ward
February 7, 2025
"Let’s shake s*** up."
Chris GeisonTheme Two Intro
March 28, 2023