Rosenverse
Hands-on AI #2: Understanding evals: LLM as a Judge

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Hands-on AI #2: Understanding evals: LLM as a Judge

Wednesday, October 15, 2025 • Rosenfeld Community

This video is featured in the Evals + Claude playlist.

Share the love for this talk
Hands-on AI #2: Understanding evals: LLM as a Judge
Speakers: Peter Van Dijck
Link:

Summary

If you’re a product manager, UX researcher, or any kind of designer involved in creating an AI product or feature, you need to understand evals. And a great way to learn is with a hands-on example.

In this second talk in the series, Peter Van Dijck of the helpful intelligence company will show you how to create an eval for an AI product using an LLM as a judge (when we use a Large Language Model to evaluate the output of another Large Language Model). We’ll have a look at how that works, but also dig into why this even works. Are we creating problems for ourselves when we let an LLM judge itself? This talk is hands on; and there will be plenty of time for questions.

You will go away understanding when and how to use LLM as a judge, and build some product sense around how the best AI products today are built, and how that can help you use them more effectively yourself.

Key Insights

  • •

    Evals are a foundational feedback loop defining what 'good' means for AI products, helping to measure and improve systems continuously.

  • •

    Evaluating fuzzy, subjective AI outputs requires innovative approaches such as using LLMs as judges to score results.

  • •

    Binary (yes/no) scoring is more reliable than rating scales with ranges because LLMs lack internal memory and consistency.

  • •

    Starting evals early (week one of a project) drastically improves AI product outcomes, but many teams delay due to perceived complexity.

  • •

    High-risk or important tasks should be prioritized for evals instead of attempting broad coverage.

  • •

    Assigning a dedicated owner or 'benevolent dictator' for evals who works closely with domain experts accelerates feedback and quality.

  • •

    Creating a written constitution of principles helps concretize AI behavior goals and guides prompt and model training.

  • •

    Most current eval tooling is too technical, slowing iteration cycles and making expert involvement inefficient.

  • •

    Custom feedback interfaces tailored to expert users significantly speed up evaluating AI outputs in domains like healthcare and law.

  • •

    Diverse perspectives from UX, product, strategy, and domain experts are critical in defining and refining what 'good' means in AI systems.

Notable Quotes

"Evals are everywhere, right? Everybody's talking about evals. It is like one of the key things in developing useful AI products."

"You want to ask an LLM to evaluate the fuzzy stuff because there’s no black and white output."

"LLMs don’t have memory, so rating on a scale from one to five is pretty random. Better to have yes or no answers."

"One of the biggest problems in AI building is evolving your prompts and having a fast feedback loop."

"By starting to categorize risk in detail, you naturally lead to better prompts and better evals."

"A constitution is a very good exercise: write down your system’s principles and values to help guide its behavior."

"Use custom systems for experts to quickly review and rate outputs, making feedback cycles much faster."

"Evals define a shared definition of good with tests to measure it, and that is the secret sauce for building great AI products."

"Model companies are students in a classroom wanting good points—they’re happy to run external expert evals to improve."

"The more I work with evals, the more I think UX and product people need to be involved because of the need for diverse perspectives."

Ask the Rosenbot
Kate Towsey
The State of ResearchOps: More Than Just Theory
2019 • DesignOps Community
Product and Design at Bloomberg: A 15-year Evolution
2022 • Design in Product 2022
Gold
Sam Proulx
SUS: A System Unusable for Twenty Percent of the Population
2021 • Civic Design 2021
Gold
Dr. Jamika D. Burge
How UX researchers can partner with (and not be replaced by) AI [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Rich Mironov
How Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
2022 • Design in Product 2022
Gold
Florence Okoye
AfroFuturism and UX Research
2023 • Advancing Research 2023
Gold
Kara Kane
Theme One Intro
2022 • Civic Design 2022
Gold
Kate Towsey
Shaping the future of research ops: Expanding roles and strategies for a more integrated research ecosystem
2025 • Advancing Research 2025
Gold
Sam Ladner
Data Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX
2016 • Enterprise UX 2016
Gold
Charlotte Vorbeck
Pipeline to Civic Design
2021 • Civic Design 2021
Gold
Louis Rosenfeld
About the Rosenverse
2026 • Rosenfeld Community
Emilia Åström
Unlock Your Team’s Intelligence with Collaboration Design
2022 • Design at Scale 2022
Gold
Erin May
Distributed, Democratized, Decentralized: Finding a Research Model to Support Your Org
2022 • Advancing Research 2022
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • Civic Design 2021
Gold
Sheryl Cababa
Living in the Clouds: Adopting a Systems Thinking Mindset
2023 • Enterprise UX 2023
Gold
Chris Moses
Stretching the Definition of DesignOps with Product Development
2018 • DesignOps Summit 2018
Gold

More Videos

Jon Fukuda

"Omni ops might be 80% standardized processes and 20% function-specific to keep necessary flexibility."

Jon Fukuda Jake Burghardt Jose Coronado Natalie Dunbar Denise Tilles

All the Ops: Successful cross-functional collaboration

September 10, 2025

Alana Washington

"We heard from the community through surveys which helped us understand what you hope to see at this year's Design Ops Summit."

Alana Washington

Theme 3 Intro

October 1, 2021

Andrew Michael

"You’ve got to take control of the backlog and position your team as value-driven, not just servicing individual teams."

Andrew Michael

Building a Product Insights Team

March 10, 2022

Nalini P. Kotamraju

"I am responsible not just for insights but for my team's well-being, legal commitments, and fiduciary responsibilities."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021

Prayag Narula

"Technologies don't have universal impact; what happens in New York vs Jakarta can be very different."

Prayag Narula Rida Qadri

HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer

March 28, 2023

Yunyan Li

"Building trust in the user is fundamental, especially in digital products like our doggy daycare app example."

Yunyan Li Anna Le Jen Kim

UX Best Practices

June 11, 2021

Shipra Kayan

"We stuck with NPS because it was a pure metric with a long history we could trend, instead of muddying things with loyalty or revenue."

Shipra Kayan

How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up

September 30, 2021

Dorelle Rabinowitz

"Debate isn’t a bad thing. Challenging each other’s thinking helps you find a better approach."

Dorelle Rabinowitz

The Magic Word is Trust

June 15, 2018

Sarah Brooks

"Anticipating and acting on risk turns uncertainty into inspiration for informed action."

Sarah Brooks

Theme 3 Intro

December 10, 2021