Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material that drastically changes UX design and requires reinventing methods and tools.
-
•
The Rose Bot chatbot was trained on 20 years of Rosenfeld Media’s content, making vast UX knowledge accessible.
-
•
Retrieval augmented generation enables chatbots to semantically search and embed large content sources for relevant answers.
-
•
Conversation logic layers complex steps like intent classification and safety filtering to make AI responses useful and appropriate.
-
•
Evaluation of generative AI systems demands substantial budget and time, often one-third of total resources, due to nondeterministic outputs.
-
•
Both human evaluation with experts and end users, alongside automated tools, are essential for assessing AI quality.
-
•
New research methods are needed for generative AI, as traditional UX research tools don’t scale or fit this new material.
-
•
Prompt deep dive technique involves detailed, think-aloud usability testing on single AI prompts to uncover nuanced interaction issues.
-
•
Co-creation between users and AI technology determines whether AI tools will be beneficial or harmful.
-
•
UX designers and product teams should embrace inventing new workflows, tooling, and evaluation practices for AI-driven products now.
Notable Quotes
"The moment GPT came out was the moment it changed for me as I saw my kids immediately adopt and use it at school."
"Generative AI is strangely good at some things and strangely bad at others, which makes it fascinating."
"Building a chatbot now is not scary, it’s just very different from how we've built things in the last 20 years."
"Evaluation takes about a third of project time and budget because you don’t know the input or output; it’s not deterministic."
"You have to evaluate conversation experiences with both experts and end users because usefulness and correctness don’t always align."
"The prompt deep dive technique has users spend 30 minutes on a single prompt with think-aloud to get deep insights."
"The future is co-created by users and technology, not driven purely by technological determinism."
"We need to invent new research methods and tooling because the material we are designing with is completely different now."
"It’s just as hard to build stuff that is bad for people as it is to build good things, so we might as well try to build good."
"We should feel okay to throw away old methods and keep fundamentals like deeply understanding users and systems."
Or choose a question:
More Videos
"Teaching product managers heuristics took 15 minutes, and now they use that shared language to articulate design."
Standardizing Product Merits for Leaders, Designers, and Everyone
June 15, 2018
"I realized I was just making it up despite the company’s growth."
Simon WardleyMaps and Topographical Intelligence (Videoconference)
January 31, 2019
"Cisgender binary options in onboarding flows exclude non-binary and gender-expansive identities."
Sandra CamachoCreating More Bias-Proof Designs
January 22, 2025
"By observing, appealing to interests, and promoting accountability, we can build empathy and respect to combat toxicity."
Darian DavisLessons from a Toxic Work Relationship
January 8, 2024
"We mitigated data privacy risks by subscribing to the corporate plan for ChatGPT, which offers enhanced security."
Fisayo Osilaja[Demo] The AI edge: From researcher to strategist
June 4, 2024
"Craft should evolve from making a beautiful final object to serving as a facilitative anchor in enterprise UX."
Uday GajendarThe Wicked Craft of Enterprise UX
May 13, 2015
"Investing in people and fostering collaboration always pays off in stronger teams and better user experiences."
Davis Neable Guy SegalHow to Drive a Design Project When you Don’t Have a Design Team
June 10, 2021
"Launching a design system is not a sprint, there’s no end, it’s always a continuous process."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"Design leaders want predictable roadmaps and healthy teams, measurable against timelines and goals."
Aurobinda Pradhan Shashank DeshpandeIntroduction to Collaborative DesignOps using Cubyts
September 9, 2022
