Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

A Mixed Method Approach to Validity to Help Build Trust
Friday, April 28, 2023 • QuantQual Interest Group (Rosenfeld Community)
Share the love for this talk
A Mixed Method Approach to Validity to Help Build Trust
Speakers: Chris Engledowl
Link:

Summary

Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.

Key Insights

  • Validity in surveys is multi-faceted, relying on five evidence sources: test content, response processes, internal structure, relation to other variables, and consequences of testing.

  • Mixed methods combining qualitative cognitive interviews and quantitative analysis enhance survey validity and build stakeholder trust.

  • Breaking down survey validation efforts across multiple teams makes the process more manageable and effective.

  • Iterative survey development over multiple rounds helps improve instrument quality while balancing the need for timely insights.

  • Qualitative research plays a vital role even within quantitative validity frameworks by revealing respondent interpretation and cognitive processes.

  • Careful stakeholder engagement and communicating rapid but incremental insights increases buy-in for rigorous validity processes.

  • Survey validity is closely tied to ethical considerations, including the impact on respondents and responsible data use.

  • Significant product or user base changes necessitate revisiting and revising surveys to maintain validity.

  • Statistical methods like factor analysis and Rasch modeling help detect underlying constructs and response biases across subpopulations.

  • It is often necessary to accept imperfect early versions of surveys, improving them progressively while acknowledging limits to change-over-time comparisons.

Notable Quotes

"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."

"Qualitative research is vital for establishing validity in mixed methods because it helps us understand how respondents interpret questions."

"Surveys are products too—they need to be iteratively tested and refined."

"You don’t know what you don’t know—surveys have blind spots that qualitative techniques can help reveal."

"If you have broader research goals shared across quant and qual teams, then the overlap supports answering difficult validity questions."

"Conversations with stakeholders need to focus on delivering usable information quickly, not just on the validity process itself."

"Consequences of testing include ethical considerations about how survey responses affect user experience and product decisions."

"It’s better to partner with quantitative experts if you don’t have that expertise yourself to understand internal structure analyses."

"If the survey doesn’t work well for subpopulations, focus initially on groups where it’s reliable while you investigate others."

"Improving surveys iteratively can undermine longitudinal measures, so it’s critical to balance validity and tracking over time."

Ask the Rosenbot
Etienne Fang
Power of Insights: Why sharing is better than silos with Uber’s Insights Platform (Videoconference)
2019 • Advancing Research Community
Scott Stephens
The Next Generation in DesignOps Toolsets (Videoconference)
2022 • DesignOps Community
Laura Smith
Embedding Service Design and Agile Practice within UK Planning Teams to Create Services that Last
2024 • Advancing Service Design 2024
Gold
George Abraham
Design Systems To-Go: Introducing a Starter Design System, and Indigo.Design Overview (Part 1)
2021 • DesignOps Summit 2021
Gold
Meghan Hellstern
The Next 100 Years of Civic Design: How Might We Better Rise to Meet the Challenges of Today and Tomorrow?
2021 • Civic Design 2021
Gold
Melinda Belcher
Bridging the Gap: Making the Most of the Differences Between Agency and Enterprise
2024 • Enterprise Experience 2020
Gold
Alan Williams
Designing essential financial services for those in need (Videoconference)
2022 • Civic Design Community
Marjorie Stainback
Transforming Strategic Research Capacity through Democratization
2019 • DesignOps Summit 2019
Gold
Mandy Drew
What Role(s) Can Research Play in Responsible Design?
2021 • Advancing Research 2021
Gold
Louis Rosenfeld
Coffee with Lou: Should You Write a (UX) Book? (Videoconference)
2024 • Rosenfeld Community
Kate Towsey
Ask Me Anything (AMA) with Kate Towsey
2025 • Rosenfeld Community
Matt Bernius
Learnings from Applying Trauma-Informed Principles to the Research Process
2022 • Advancing Research 2022
Gold
Operationalizing DesignOps
2018 • DesignOps Summit 2018
Gold
Dave Malouf
The Past, Present, and Future of DesignOps: a 2-part DesignOps Community Call (Part 2) (Videoconference)
2022 • DesignOps Community
Alastair Simpson
Debunking the Myths of Cross-Disciplinary Collaboration
2019 • DesignOps Summit 2019
Gold
Courtney Kaplan
Taking it to the next level: Career paths in DesignOps
2018 • DesignOps Summit 2018
Gold

More Videos

Zariah Cameron

"Burnout really exists because we made rest a reward rather than a right."

Zariah Cameron

ReDesigning Wellbeing for Equitable Care in the Workplace

September 23, 2024

John Cutler

"Priorities aren’t priorities until they’ve been prioritized with attributes and values."

John Cutler Harry Max

Prioritization for designers and product managers (1st of 3 seminars) (Videoconference)

June 13, 2024

Nick Cochran

"Every time somebody purchased a domain name, I earned a commission. Over three to four years, I earned over $100,000 as a high school student."

Nick Cochran

Growing in Enterprise Design through Making Connections

June 3, 2019

Deanna Smith

"You should assess feedback before making decisions; implementing feedback right away can cause the wrong solution or disruptions."

Deanna Smith

Leading Change with Confidence: Strategies for Optimizing Your Process

September 23, 2024

Magdalena Zadara

"We’re now looking to help legislators think about digital outcomes as they draft new laws and regulations."

Magdalena Zadara

Zero Hour: How to Get Far Quickly When Starting Your Digital Service Unit Late

November 16, 2022

Nova Wehman-Brown

"Despite all the insecurities, most team members were willing to show up every week and challenge what they thought they knew."

Nova Wehman-Brown

We've Never Done This Before

June 4, 2019

Liwei Dai

"After more than 10 years of doing UX research, I was still only working above the iceberg, focusing on usability and feature level."

Liwei Dai

The Heart and Brain of the AI Research

March 31, 2020

Gordon Ross

"The DDN shortened the network distance between any two government staff who found themselves thrown into COVID response work, making coordination faster."

Gordon Ross

12 Months of COVID-19 Design and Digital Response with the British Columbia Government

December 8, 2021

Tutti Taygerly

"Listening to the 10% truth means acknowledging a bit of what they say even if you’re irritated."

Tutti Taygerly

Make Space to Lead

June 12, 2021