Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.
Key Insights
-
•
Validity in surveys is multi-faceted, relying on five evidence sources: test content, response processes, internal structure, relation to other variables, and consequences of testing.
-
•
Mixed methods combining qualitative cognitive interviews and quantitative analysis enhance survey validity and build stakeholder trust.
-
•
Breaking down survey validation efforts across multiple teams makes the process more manageable and effective.
-
•
Iterative survey development over multiple rounds helps improve instrument quality while balancing the need for timely insights.
-
•
Qualitative research plays a vital role even within quantitative validity frameworks by revealing respondent interpretation and cognitive processes.
-
•
Careful stakeholder engagement and communicating rapid but incremental insights increases buy-in for rigorous validity processes.
-
•
Survey validity is closely tied to ethical considerations, including the impact on respondents and responsible data use.
-
•
Significant product or user base changes necessitate revisiting and revising surveys to maintain validity.
-
•
Statistical methods like factor analysis and Rasch modeling help detect underlying constructs and response biases across subpopulations.
-
•
It is often necessary to accept imperfect early versions of surveys, improving them progressively while acknowledging limits to change-over-time comparisons.
Notable Quotes
"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."
"Qualitative research is vital for establishing validity in mixed methods because it helps us understand how respondents interpret questions."
"Surveys are products too—they need to be iteratively tested and refined."
"You don’t know what you don’t know—surveys have blind spots that qualitative techniques can help reveal."
"If you have broader research goals shared across quant and qual teams, then the overlap supports answering difficult validity questions."
"Conversations with stakeholders need to focus on delivering usable information quickly, not just on the validity process itself."
"Consequences of testing include ethical considerations about how survey responses affect user experience and product decisions."
"It’s better to partner with quantitative experts if you don’t have that expertise yourself to understand internal structure analyses."
"If the survey doesn’t work well for subpopulations, focus initially on groups where it’s reliable while you investigate others."
"Improving surveys iteratively can undermine longitudinal measures, so it’s critical to balance validity and tracking over time."
Dig deeper—ask the Rosenbot:
More Videos
"We scaled panels down to about 60 countries in 11 languages focusing on markets important to Airbnb teams."
Wyatt HaymanGlobal Research Panels (Videoconference)
August 8, 2020
"We investigate whether we can skin native components to respect platform conventions while identifying commonalities and differences."
PJ Buddhari Nate BaldwinMeet Spectrum, Adobe’s Design System
June 9, 2021
"Exploring Wi-Fi-free zones in Toronto highlights citizen-driven desires to unplug and maintain data privacy in public spaces."
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"Middle managers are responsible for the how—process, coordination, and communication—and you don’t see the value of that until it’s missing."
Peter MerholzThe Trials and Tribulations of Directors of UX (Videoconference)
July 13, 2023
"Feeling heard and seen by research practitioners is one of the best compliments a participant can give."
Dr. Jamika D. Burge Mansi GuptaAdvancing the Inclusion of Womxn in Research Practices (Videoconference)
September 15, 2022
"I never want to walk into a review with my stakeholders and see something for the first time."
Amy MarquezINVEST: Discussion
June 15, 2018
"Do you play games? Cool, you’re in. Stop asking questions. It’s about behavior, not identity."
Dane DeSutter Natalie Gedeon Deborah Hendersen Cheryl PlatzBeyond the Console: The rise of the Gamer Experience and how gaming will impact UX Research across industries (Videoconference)
May 17, 2024
"If I don’t trust you, how can I feel safe with you?"
Zariah CameronReDesigning Wellbeing for Equitable Care in the Workplace
September 23, 2024
"Breaking tasks down into manageable chunks creates dopamine hits and makes work less overwhelming."
Jessica NorrisADHD: A DesignOps Superpower
September 9, 2022
