Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Building the Rosenbot

Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
Building the Rosenbot
Speakers: Peter Van Dijck
Link:

Summary

A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?

Key Insights

  • Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.

  • Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.

  • Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.

  • Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.

  • Effective evaluation requires combining human expert and end-user feedback alongside automated tools.

  • A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.

  • ‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.

  • New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.

  • UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.

  • Co-creation between users and technology defines how AI applications evolve and succeed or fail.

Notable Quotes

"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."

"This AI stuff is a new design material, just like the internet or mobile was before."

"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."

"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."

"There’s a lot of steps behind the scenes to make conversations useful and safe."

"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."

"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."

"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."

"Co-creation between users and technology determines if this AI evolves to be useful or harmful."

"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."

Ask the Rosenbot
Luke Wroblewski
Designing AI Applications
2026 • Rosenfeld Community
Chris Chapo
Data Science and Design: A Tale of Two Tribes
2015 • Enterprise UX 2015
Gold
Kristin Sundermeyer
Design Ops Metrics
2021 • DesignOps Summit 2021
Gold
Mariesa Lenz
What Beekeeping Taught me about Product Teams
2025 • Rosenfeld Community
Dr. Jamika D. Burge
The Future of Research: Bridging the Gaps
2021 • Advancing Research Community
Alberto Ferreira
Making it Count: Developing a custom digital metric framework that works
2021 • QuantQual Interest Group
Patrizia Bertini
DesignOps + KPIs = Measure your Impact!
2024 • DesignOps Summit 2020
Gold
Saara Kamppari-Miller
Inclusive Design is DesignOps
2021 • DesignOps Summit 2021
Gold
Dianne Que
Real Talk: Proving Value through a Scrappy Playbook
2019 • DesignOps Summit 2019
Gold
Chris Govias
Perspectives on Civic Design
2021 • Civic Design Community
Emily Danielson
“I mean, I can lift a shovel”: Design Skills in Disaster Response
2022 • Design at Scale 2022
Gold
Ali Jeffery
How DesignOps Helped Enable Wall Street to Work Remotely
2020 • DesignOps Summit 2020
Gold
Sheryl Cababa
Living in the Clouds: Adopting a Systems Thinking Mindset
2023 • Enterprise UX 2023
Gold
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Samuel Proulx
Inclusive Research: Debunking Myths and Getting Started
2025 • Advancing Research 2025
Gold
Jay Bustamante
Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation
2023 • DesignOps Summit 2023
Gold

More Videos

Xenia Adjoubei

"The use of dovetail helps organize diverse information without losing the author’s voice or connection to original data."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Tamara Kartoziia

"My definition of service design is dancing with needs and value."

Tamara Kartoziia

Think global, adapt local: how service design accelerated B2B market entry by 6 months

November 20, 2025

Jay Bustamante

"Your role is to ensure that the right people are at the party — a multidisciplinary team."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Nicole Aleong

"There are over 7 billion futures of right now."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Nick Lewis

"Generative AI is not creative; it’s just pattern finding and recycling what’s been done before."

Nick Lewis

Designing and building low-carbon websites independently

November 18, 2025

Vicky Teinaki

"Even if you had designed it, I wouldn’t have prioritized it."

Vicky Teinaki Michele Marut Tim Parmee

Short Take #3: UX/Product Lessons from Your Industry Peers

December 6, 2022

Laura Weiss

"Conflict resolution skills include self-regulation, acknowledgment, asking exploratory questions, and deep listening."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Joerg Beringer

"Our vision was to create an AI first tool for user research that delivers instantly well structured requirements and design inputs."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

June 10, 2025

Shelby Switzer

"Zoom fatigue is real, people are not necessarily keen to go into another Zoom meeting even if they think it’s valuable."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021