Rosenverse
Building impactful AI products for design and product leaders, Part 2: Evals are your moat

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Building impactful AI products for design and product leaders, Part 2: Evals are your moat

Wednesday, July 23, 2025 • Rosenfeld Community

This video is featured in the AI and UX playlist.

Share the love for this talk
Building impactful AI products for design and product leaders, Part 2: Evals are your moat
Speakers: Peter Van Dijck
Link:

Summary

The secret ingredient for impactful AI products is “evals”—an architecture for ongoing evaluation of quality. Without evals, you don’t know if your output is good. You don’t know when you’re done. Because outputs are non-deterministic, it’s very hard to figure out if you are creating real value for your users, and when something goes wrong, it’s really tricky to figure out why.

Simply Put’s Peter van Dijck will demystify evals, and share a simple framework for planning for and building useful evals, from qualitative user research to automated evals using LLMs as a judge.

Key Insights

  • •

    AI product development involves three layers: model capabilities, context management, and user experience, with evals central to experience quality assurance.

  • •

    Automated evals help scale testing of AI with inherently open-ended inputs and outputs, enabling faster iteration cycles with confidence.

  • •

    LLMs can serve as judges (evaluators) of other LLM outputs, which works because classification is cognitively easier than generation.

  • •

    Defining what 'good' means for an AI system is a detailed, evolving process informed by research, domain expertise, and observed risks.

  • •

    A three-option evaluation (e.g., yes/no/maybe) works better than fine-grained scales for consistent automated scoring by LLMs.

  • •

    Synthetic data, generated by LLMs based on manually created examples, efficiently expands dataset breadth and usefulness.

  • •

    Domain experts are essential for tagging data and establishing quality criteria, especially for high-stakes areas like healthcare or legal.

  • •

    Building effective evals requires substantial effort—expect 20-40% of project resources devoted to this work.

  • •

    Cultural differences impact subjective evals like politeness, requiring localization and careful domain definition.

  • •

    AI product quality management is a strategic ongoing commitment, extending beyond initial development into production monitoring and iteration.

Notable Quotes

"AI products almost always have both open-ended inputs and outputs, which makes testing really hard."

"You have to build a detailed definition of what is good for my system to do meaningful automated evals."

"It’s much easier to classify an answer than to generate an answer, and that’s why LLM as a judge works."

"You don’t want to give too many options like rating from one to ten because consistency gets lost between different LLM calls."

"Synthetic data is useful because it’s easier to generate more examples of something you already have than to create entirely new data."

"If you launch in the US and politeness is an issue, first try to fix it with prompts; only if that fails should you build an eval."

"Evals are really your intellectual property—they define what good looks like in your domain."

"Domain experts are crucial for tagging data because users might say ‘that’s great,’ but experts can tell it’s totally wrong."

"You should plan 20 to 40 percent of your project budget on evals—it’s a lot more work than most people expect."

"This is where UX and product strategy bring huge value—defining what good means rather than leaving it to engineers alone."

Ask the Rosenbot
Bria Alexander
Opening Remarks
2022 • DesignOps Summit 2022
Gold
Tutti Taygerly
Make Space to Lead
2021 • Design at Scale 2021
Gold
Ned Gartside
Navigating accessibility and climate
2024 • Climate UX Interest Group
Sam Ladner
How Research Can Drive Strategic Foresight
2022 • Advancing Research 2022
Gold
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps Summit 2024
Gold
Christopher Taylor Edwards
Design as a Team Practice, A Practical Guide to Cross-functional Collaboration
2021 • DesignOps Summit 2021
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Audrey Crane
Shadow Design–Where Else is Design Happening in Your Organization?
2023 • Enterprise Community
Laureen Kattan
Centering Patients and Clinicians in a Complex Government Ecosystem
2023 • Design in Product 2023
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • DesignOps Summit 2021
Gold
Sara Asche Anderson
Not Your Ordinary Re-Brand: Design's Path to Driving Customer Obsession at Best Buy
2024 • Enterprise Experience 2020
Gold
Bilan Hashi
The Tension Between Story Collecting and Story Telling in Research
2021 • Advancing Research 2021
Gold
Joseph Williams
Unlocking impact and influence through inclusive hiring in research
2021 • Advancing Research Community
Tatyana Mamut
Opening Keynote: Breaking Conway's Law--or How to Work Differently and Not Ship Your Org Chart
2019 • Enterprise Experience 2019
Gold
Kristin Skinner
Group Activity: A Deep Dive Into Value and Outcomes
2019 • DesignOps Summit 2019
Gold
Russ Unger
Onboarding: The Ecosystem, not the Afterthought
2017 • DesignOps Summit 2017
Gold

More Videos

Simon Wardley

"Political capital is critical; good strategy without support will be crushed in large organizations."

Simon Wardley

Maps and Topographical Intelligence

January 31, 2019

Megan Blocker

"Knowledge just isn’t created in isolation anymore."

Megan Blocker

Theme 2 Intro

March 12, 2025

Mila Kuznetsova

"Kids are not little adults. You can’t just simplify the text and call it for kids."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Megan Blocker

"Maturity is not just knowing what to do, but also knowing what not to do and being smart and grown up about saying no."

Megan Blocker Lada Gorlenko Fatimah Richmond Molly Stevens

What UX research maturity looks like and how we get there [Advancing Research Community Workshop Series]

November 9, 2023

Wendy Johansson

"Almost every kind of UX is happening in settings that have big challenges of scale and complexity that you see in enterprises."

Wendy Johansson Surya Vanka

Design at Scale: Behind the Scenes

April 29, 2021

David Sternberg

"Behavioral turbulence happens when flow breaks down, causing chaotic and unpredictable user interactions."

David Sternberg

Uncovering the hidden forces shaping user behavior

July 17, 2025

Sam Proulx

"If it’s not sold at Amazon or Costco, it’s probably not in my house — not because I love those companies but because of their consistency."

Sam Proulx

Online Shopping: Designing an Accessible Experience

November 29, 2023

John Cutler

"Sometimes leaders focus so much on scanning the river they’re not observing the surroundings—the banks tell you if the river is deep or shallow."

John Cutler

Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)

December 3, 2024

Jayne Engle

"The seventh generation principle calls for long-term decision making that transcends self-interest."

Jayne Engle Tanya Chung-Tiam-Fook

Civic Design for the Next Seven Generations—A Discussion on Sacred Civics

August 25, 2022