Building the Rosenbot
Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"When projects are late or delayed it’s often due to organizational and cultural issues rather than conduct of the project team."
Carl TurnerYou Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
March 28, 2023
"You earn your seat at the table by championing excellent customer experience and rallying the project team behind it."
Karen PascoeDeveloping Experience Teams and Talent in the Enterprise
June 8, 2016
"We couldn’t find just one word to describe our third value so we made one up: humbladent—humble and confident together."
Jilanna WilsonDistributed Design Operations Management
October 23, 2019
"Designers contribute experimental patterns back to the design system early, making their work visible even before full validation."
Shawna Hein Kevin HoffmanCreate a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts
November 17, 2022
"Site visits with users don’t just give actionable insight; they give us raw, humbling inspiration to keep going."
Barb SpantonDoing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
June 10, 2022
"As the delivery driver, I connected the dots, got people briefs, and sometimes said no to lofty leadership requests."
Briana ThomasThe Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
September 24, 2024
"Therapists create space for clients to reflect and discover insights in a safe environment."
Gina MendoliaTherapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
December 3, 2024
"If you press the exact same button today and tomorrow you get two different results, that’s non-deterministic UX."
Lija Hogan Milan Mijatovic Sam Proulx Louis RosenfeldThree Years Out: Perspectives on the Near-Term Future of User Research
March 15, 2024
"Imagine your current design ops team is like a coin and every time you flip it, it comes up heads, but coins have two sides."
John Calhoun Rachel PosmanTwo Sides of the DesignOps Coin: Teams Ops and Product Ops
January 8, 2024
