Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Garbage in, garbage out? Measuring error rates to get ready for AI

Thursday, January 8, 2026 • Rosenfeld Community
Share the love for this talk
Garbage in, garbage out? Measuring error rates to get ready for AI
Speakers: Caroline Jarrett
Link:

Summary

We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it. Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI? In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways: 1) Problems along the way 2) Wrong result 3) Unnecessary action 4) Delayed-impact problem 5) Non-uptake or over-uptake 6) Technology problem We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.

Key Insights

  • Errors in data collection and user input are foundational issues that compromise AI and service outcomes.

  • Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.

  • Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.

  • Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.

  • Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.

  • Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.

  • Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.

  • Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.

  • AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.

  • Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.

Notable Quotes

"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."

"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."

"Sometimes a form forces you into a wrong answer by giving inappropriate options."

"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."

"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."

"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."

"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."

"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."

"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Ask the Rosenbot
Ali Jeffery
How DesignOps Helped Enable Wall Street to Work Remotely
2020 • DesignOps Summit 2020
Gold
Audrey Crane
Shadow Design–Where Else is Design Happening in Your Organization?
2023 • Enterprise Community
Sean McKay
Whole Product Thinking: Expanding beyond problem and solution space thinking
2024 • Rosenfeld Community
Julie Gitlin
Design as an Agent of Digital Transformation at JPMC
2021 • Design at Scale 2021
Gold
Randolph Duke II
War Stories LIVE! Randy Duke II
2020 • Advancing Research 2020
Gold
David Cronin
Discussion
2015 • Enterprise UX 2015
Gold
Steve Portigal
War Stories LIVE! Steve Portigal
2020 • Advancing Research 2020
Gold
Nicole Wright
Democratizing Research at HoneyBook
2022 • Advancing Research 2022
Gold
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Design at Scale 2022
Gold
Sylvie Abookire
A Civic Designer's Guide to Mindful Conflict Navigation
2022 • Civic Design 2022
Gold
Daniel Korczynski
Why AI Is Bad at Research (and how to make it actually useful)
2026 • Advancing Research 2026
Gold
Anna Avrekh
User Research, Design, and Product - A Love Story
2021 • Advancing Research 2021
Gold
Jules Monza
Use These Words and Count These Things
2024 • DesignOps Summit 2024
Gold
Husani Oakley
Theme Three Intro
2023 • Enterprise UX 2023
Gold
Joshua Graves
We Need To Talk: Addressing Unmet Expectations (Part 2 of 3)
2025 • Rosenfeld Community
Clara Kliman-Silver
UX Futures: The Role of Artificial Intelligence in Design
2023 • Enterprise UX 2023
Gold

More Videos

Leisa Reichelt

"If you make rules, put numbers in them, and say them confidently, sometimes people actually do them."

Leisa Reichelt

Opening Keynote: Operating in Context

November 7, 2018

Cennydd Bowles

"There is no value in outsourcing the thinking of philosophy to AI; doing the thinking yourself is the point."

Cennydd Bowles

Exit Interview #2: Rediscovering the ethical heart of design

November 6, 2025

Andrew Michael

"Marketing is an essential collaborator because they help package and communicate insights organization-wide."

Andrew Michael

Building a Product Insights Team

March 10, 2022

Veevi Rosenstein

"Since everyone was doing their own research, the quality of the research being done was pretty inconsistent."

Veevi Rosenstein

Building for Scale: Creating the Zendesk UX Research Practice

January 8, 2024

Megan Campos

"We often center the point of view of dominant demographics and ignore the needs of those we never speak to."

Megan Campos

What Did I Miss? The Hidden Costs of Deprioritizing Diversity in User Research

March 12, 2021

Aras Bilgen

"AI users have reduced critical thinking skills within just two years of using AI-assisted tools."

Aras Bilgen

Who does the math: A designer’s journey in building an AI-based tutoring app

June 10, 2025

Ron Bronson

"That friends is service experience in the 21st century."

Ron Bronson

Design, Consequences & Everyday Life

November 18, 2022

Tina Weisser

"We must design who is in the loop and who is in the lead. The responsibility can’t be handed over to AI."

Tina Weisser

When AI Agents Meet Reality. Service Design Lessons from a Pilot

February 26, 2026

Dr Chloe Sharp

"For bigger decisions, having documented meetings was critical for transparency and defending choices later on."

Dr Chloe Sharp

Using Evidence and Collaboration for Setting and Defending Priorities

November 29, 2023