Building the Rosenbot
Summary
- What does strategy look like in an AI world?
- What does eval-first mean?
- What the hell is going on deep inside an LLM?
- And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"Empathy applies to us too—we needed to understand how our own people were feeling about these layered changes."
Erin HauberDesign is Not the Frosting on the Scaled Agile Layer Cake
October 24, 2019
"The only person who gets to decide what it means to be badass is you."
Megan KiersteadYou Are a Badass at UX: Overcoming Imposter Syndrome
March 10, 2021
"Working at Uber requires grit."
Maggie DieringerCreating Consistency Through Constant Change
January 8, 2024
"Lift and shift equals timidity—it’s sticking to what you know rather than delivering real value."
Meaghan Waters Fotina KoutropoulousLack of Product Thinking will Doom Your Legacy Modernization
June 9, 2021
"Market research can no longer afford to take six months with multiple waves; it has to be speedier and more agile now."
Jemma AhmedBringing together market and user research
October 17, 2019
"Most of institutional knowledge is tacit, meaning it’s understood or implied without being stated."
Maria TaylorKnowledge is Power: Managing the Lifeblood of the Design Org
October 3, 2023
"Getting accessibility integrated into all company processes is essential because accessibility is a culture change."
Sam ProulxUnderstanding Screen Readers on Mobile: How And Why to Learn from Native Users
June 6, 2023
"We were spending 90% of our resource responding to tactical queries or performance reporting — essentially marking everyone's homework."
Catherine BlizzardUsing Integrated Insight to Drive Growth
March 10, 2022
"Carol was really grateful for our conversation."
Randolph Duke IIWar Stories LIVE! Randy Duke II
March 30, 2020
