Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

A Mixed Method Approach to Validity to Help Build Trust

Friday, April 28, 2023 • QuantQual Interest Group
Share the love for this talk
A Mixed Method Approach to Validity to Help Build Trust
Speakers: Chris Engledowl
Link:

Summary

Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.

Key Insights

  • Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.

  • Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.

  • Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.

  • Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.

  • Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.

  • Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.

  • Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.

  • Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.

  • Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.

  • It is important to revisit validity considerations continuously, especially after product or user base changes.

Notable Quotes

"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."

"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."

"What would happen if you do think about validity? Would it change your process or research plans?"

"Surveys are products too, so they need to be iteratively tested."

"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."

"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."

"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."

"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."

"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."

"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."

Ask the Rosenbot
Rich Mironov
How Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
2022 • Design in Product 2022
Gold
Frances Yllana
The Big Question about Impact: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Clara Kliman-Silver
UX Futures: The Role of Artificial Intelligence in Design
2023 • Enterprise UX 2023
Gold
Sam Proulx
Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
2022 • Civic Design 2022
Gold
Jon White
Unsticking Research for Better Information Flow
2026 • Advancing Research 2026
Gold
Joshua Graves
We Need To Talk: Navigating Conversations with Your Boss (Part 1 of 3)
2025 • Rosenfeld Community
Dan Ward
Failure Friday #1 with Dan Ward
2025 • Rosenfeld Community
Josh Clark
Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
2025 • Rosenfeld Community
Alexandra Schmidt
Why Ethics Can't Save Tech
2022 • Civic Design 2022
Gold
Josh Clark
Sentient Design: New Design Patterns for New Experiences (3rd of 3 seminars)
2025 • Rosenfeld Community
Alana Washington
Theme 1: Introduction and Provocation
2024 • DesignOps Summit 2020
Gold
Rob Mitzel
The Tale of Two Companies: Building a Successful UX Practice in a Century-Old Enterprise
2024 • Enterprise Experience 2020
Gold
Dan Willis
Enterprise Storytelling Sessions
2015 • Enterprise UX 2015
Gold
Suzan Bednarz
AccessibilityOps for All
2024 • DesignOps Summit 2020
Gold
Darian Davis
Lessons from a Toxic Work Relationship
2024 • Enterprise Experience 2020
Gold
Melissa Schmidt
How UX Research Hit It Big in Las Vegas
2019 • Enterprise Experience 2019
Gold

More Videos

John Cutler

"Some of the most important decisions for service design happen on whiteboards among engineers, often without service design representation."

John Cutler

Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)

December 3, 2024

Darian Davis

"Taking responsibility for our behaviors starts with an apology and continues with regular feedback and action."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Dave Gray

"You don’t have to sit through two hours a day; we’ll spend about 15 minutes on setup then you’ll pair up and do the exercise."

Dave Gray

Group Activity: Making Sense of DesignOps

November 7, 2017

John Cutler

"A story lets people assign the meaning that's powerful to them and then think about the decisions they need to make and align."

John Cutler

The Alignment Trap

November 29, 2023

Abby Covert

"Providing alternative content is often a kinder gesture than any lengthy alternative text could be."

Abby Covert

Stuck? Diagrams Help

October 27, 2022

Mark Interrante

"Use four simple questions to find out what people really want and why — it avoids misaligned efforts."

Mark Interrante

Collaboration Flows in Product Development

June 9, 2017

Devon Powers

"The future will be struggle; we need to face that forward, not deny it."

Devon Powers

Imagining Better Futures

March 9, 2022

Prayag Narula

"Design research—they need each other. They are much, much better together."

Prayag Narula

How to Empower Your Designers to Do Good Research – And Why You Want To

June 10, 2022

Peter Van Dijck

"If you build for the next model, even if it can hardly do something now, it will probably do it well in the future."

Peter Van Dijck

Hands on AI #3: Claude Code for UX people

October 22, 2025