Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material that drastically changes UX design and requires reinventing methods and tools.
-
•
The Rose Bot chatbot was trained on 20 years of Rosenfeld Media’s content, making vast UX knowledge accessible.
-
•
Retrieval augmented generation enables chatbots to semantically search and embed large content sources for relevant answers.
-
•
Conversation logic layers complex steps like intent classification and safety filtering to make AI responses useful and appropriate.
-
•
Evaluation of generative AI systems demands substantial budget and time, often one-third of total resources, due to nondeterministic outputs.
-
•
Both human evaluation with experts and end users, alongside automated tools, are essential for assessing AI quality.
-
•
New research methods are needed for generative AI, as traditional UX research tools don’t scale or fit this new material.
-
•
Prompt deep dive technique involves detailed, think-aloud usability testing on single AI prompts to uncover nuanced interaction issues.
-
•
Co-creation between users and AI technology determines whether AI tools will be beneficial or harmful.
-
•
UX designers and product teams should embrace inventing new workflows, tooling, and evaluation practices for AI-driven products now.
Notable Quotes
"The moment GPT came out was the moment it changed for me as I saw my kids immediately adopt and use it at school."
"Generative AI is strangely good at some things and strangely bad at others, which makes it fascinating."
"Building a chatbot now is not scary, it’s just very different from how we've built things in the last 20 years."
"Evaluation takes about a third of project time and budget because you don’t know the input or output; it’s not deterministic."
"You have to evaluate conversation experiences with both experts and end users because usefulness and correctness don’t always align."
"The prompt deep dive technique has users spend 30 minutes on a single prompt with think-aloud to get deep insights."
"The future is co-created by users and technology, not driven purely by technological determinism."
"We need to invent new research methods and tooling because the material we are designing with is completely different now."
"It’s just as hard to build stuff that is bad for people as it is to build good things, so we might as well try to build good."
"We should feel okay to throw away old methods and keep fundamentals like deeply understanding users and systems."
Or choose a question:
More Videos
"I wish I could tell my 22-year-old self what to stop doing and what to embrace to be a better designer."
Adam Cutler Karen Pascoe Ian Swinson Susan WorthmanDiscussion
June 8, 2016
"UXers are less satisfied than their peers because we have failed to set expectations about the real work of UX in organizations."
Peter MerholzThe Trials and Tribulations of Directors of UX (Videoconference)
July 13, 2023
"Governance isn’t about choking creativity, it’s about putting bounds and clarity that enable purposeful collaboration."
Lisa WelchmanCleaning Up Our Mess: Digital Governance for Designers
June 14, 2018
"Sustainable practices are not just a luxury; they are a necessity for our survival."
Vincent BrathwaiteOpener: Past, Present, and Future—Closing the Racial Divide in Design Teams
October 22, 2020
"OKRs are a tool for each of us to tidy our house and focus on what’s important."
Brenna FallonLearning Over Outcomes
October 24, 2019
"Hip hop proves that we can re-animate spaces with highly generative communities that weren't built for us."
Tricia WangSpatial Collapse: Designing for Emergent Culture
January 8, 2024
"If you don’t bring stakeholders into the research journey, they won’t believe or use the data."
Edgar Anzaldua MorenoUsing Research to Determine Unique Value Proposition
March 11, 2021
"Our brains are terrible at operating only on one type of information, whether object-oriented or context-oriented."
Designing Systems at Scale
November 7, 2018
"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."
Erin WeigelGet Your Whole Team Testing to Design for Impact
July 24, 2024
