Rosenverse
Garbage in, garbage out? Measuring error rates to get ready for AI

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Garbage in, garbage out? Measuring error rates to get ready for AI

Thursday, January 8, 2026 • Rosenfeld Community
Share the love for this talk
Garbage in, garbage out? Measuring error rates to get ready for AI
Speakers: Caroline Jarrett
Link:

Summary

We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it.

Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI?

In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways:
1) Problems along the way
2) Wrong result
3) Unnecessary action
4) Delayed-impact problem
5) Non-uptake or over-uptake
6) Technology problem
We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.

Key Insights

  • •

    Errors in data collection and user input are foundational issues that compromise AI and service outcomes.

  • •

    Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.

  • •

    Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.

  • •

    Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.

  • •

    Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.

  • •

    Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.

  • •

    Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.

  • •

    Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.

  • •

    AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.

  • •

    Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.

Notable Quotes

"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."

"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."

"Sometimes a form forces you into a wrong answer by giving inappropriate options."

"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."

"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."

"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."

"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."

"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."

"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Ask the Rosenbot
Verónica Urzúa
The B-side of the Research Impact
2021 • Advancing Research 2021
Gold
Mackenzie Guinon
M.C. Escher’s UX Research Career Ladder
2022 • Advancing Research 2022
Gold
Chris Geison
What's Next for Research?
2021 • Advancing Research Community
Ellen Chisa
The Values of Design
2023 • Design in Product 2023
Gold
Davis Neable
How to Drive a Design Project When you Don’t Have a Design Team
2021 • Design at Scale 2021
Gold
Nicole Umphress
Delivering Design Education During a Global Pandemic: Lessons Learned
2022 • Design at Scale 2022
Gold
Ariel Kennan
Building a Design Culture
2017 • Enterprise Experience 2017
Gold
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Jim Kalbach
Jobs To Be Done
2021 • Enterprise Community
Charlotte Lee
Theme 1 Intro
2021 • Civic Design 2021
Gold
Katy Mogal
But Do Your Insights Scale?
2021 • Advancing Research 2021
Gold
Harry Brignull
Beyond Clicks and Tricks: Why deceptive design has grown into a regulatory faultline
2026 • Rosenfeld Community
Dr. Jamika D. Burge
How UX researchers can partner with (and not be replaced by) AI [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Michelle Morrison
Practice What You Preach
2024 • DesignOps Summit 2020
Gold
Kelly Goto
Emotion Economy: Ethnography as Corporate Strategy
2015 • Enterprise UX 2015
Gold
Bria Alexander
Day 1 Panel: Up to the Minute: The latest in AI’s impact on UX
2025 • Designing with AI 2025
Gold

More Videos

Simon Wardley

"Very few people in business are able to understand or familiar with the concept of situational awareness unless they served in the military."

Simon Wardley

Maps and Topographical Intelligence

January 31, 2019

Megan Blocker

"If we want research to thrive, we need to equip ourselves with more tools, not fewer."

Megan Blocker

Theme 2 Intro

March 12, 2025

Mila Kuznetsova

"Power dynamics exist in every session. People don’t want to be embarrassed or feel put on the spot."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Megan Blocker

"Creating personas as a team is more valuable than the persona itself because it creates ownership and empathy."

Megan Blocker Lada Gorlenko Fatimah Richmond Molly Stevens

What UX research maturity looks like and how we get there [Advancing Research Community Workshop Series]

November 9, 2023

Wendy Johansson

"Almost every kind of UX is happening in settings that have big challenges of scale and complexity that you see in enterprises."

Wendy Johansson Surya Vanka

Design at Scale: Behind the Scenes

April 29, 2021

David Sternberg

"The interface you’ve designed is like a landscape shaping the flow of user behavior as water flows through a stream."

David Sternberg

Uncovering the hidden forces shaping user behavior

July 17, 2025

Sam Proulx

"Convenience includes accepting multiple payment providers like Apple Pay or PayPal, which can mean the difference between completing and abandoning a purchase."

Sam Proulx

Online Shopping: Designing an Accessible Experience

November 29, 2023

John Cutler

"It is impossible to survive in this industry by pushing against the current all the time; you must put yourself in positions to leverage the current."

John Cutler

Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)

December 3, 2024

Jayne Engle

"Proofs of possibility, not just experiments, are essential to demonstrate viable pathways for systemic transformation."

Jayne Engle Tanya Chung-Tiam-Fook

Civic Design for the Next Seven Generations—A Discussion on Sacred Civics

August 25, 2022