Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?
Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be linked to clear business imperatives and KPIs to avoid meaningless data collection.

  • Statistical significance alone does not determine if a finding is meaningful to business strategy.

  • Sample size and data variability significantly impact whether differences in scores are statistically significant.

  • Qualitative research provides crucial context and meaning that pure quantitative methods often miss.

  • Competitive benchmarking should include industries outside your own to meet evolving customer experience expectations.

  • Significant changes parallel to industry trends may not represent meaningful progress for a company.

  • Driver modeling helps prioritize research efforts by identifying which perceptions most influence key outcomes like NPS or loyalty.

  • Frequent measurement without implementing changes dilutes insights and wastes resources.

  • Researchers should tailor findings to the priorities of different stakeholders to increase buy-in and understanding.

  • A mixed-methods approach combining qualitative and quantitative data yields the most actionable and trustworthy insights.

Notable Quotes

"Research is not easy to do and it’s not something we should waste time in; it needs to be well thought out."

"A number like an NPS score means different things to different people depending on their role."

"Sample size has a direct impact on whether a change in score is statistically significant or just noise."

"Just because a change is statistically significant doesn’t mean it’s meaningful enough to change business strategy."

"When the pandemic hit, loyalty was shattered across industries; companies had to earn it back."

"If you measure without making changes, you shouldn’t be measuring more often — that’s a big no-no."

"If you had an infinite sample size, every change would be significant, which defeats the purpose."

"Driver modeling tells you which initiatives matter most so you know where to focus your research efforts."

"Attach your findings to what executives care about — increasing revenue, decreasing costs, innovating faster."

"I always do mixed methods because one alone doesn’t give you everything to truly understand what’s going on."

Ask the Rosenbot
Billy Carlson
Tips to Utilize Wireframes to Tell an Effective Product Story
2023 • Enterprise UX 2023
Gold
Sean Fitzell
Craft of User Research: Building Out Jobs to be Done Maps
2021 • Advancing Research 2021
Gold
Uday Gajendar
From AI to Zeitgeist: Theory as the design antidote to AI hype
2025 • Rosenfeld Community
Nalini P. Kotamraju
An Organizational Story: Salesforce Lightning Design System
2016 • Enterprise UX 2016
Gold
Chris Govias
Perspectives on Civic Design (Videoconference)
2021 • Civic Design Community
Maggie Dieringer
Creating Consistency Through Constant Change
2024 • DesignOps Summit 2020
Gold
Kayla Farrell
What It's Like To Be a User Researcher at Compass
2021 • Advancing Research 2021
Gold
Robin Beers
Panel: Excellence in Communicating Insights
2024 • Advancing Research 2024
Gold
George Abraham
Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)
2021 • DesignOps Summit 2021
Gold
Ted Neward
Theme 4: Enterprise Organizational Journey
2019 • Enterprise Experience 2019
Gold
PJ Buddhari
Meet Spectrum, Adobe’s Design System
2021 • Design at Scale 2021
Gold
Holly Cole
Understanding Experiences: When you have to do more than work
2018 • DesignOps Summit 2018
Gold
Shahrzad Samadzadeh
What Is My Value? Two Takes and Some Mistakes
2024 • Enterprise Experience 2020
Gold
Dawn Ressel
Full-Stack User Experiences: A Marriage of Design and Technology
2016 • Enterprise UX 2016
Gold
Isaac Heyveld
Expand DesignOps Leadership as a Chief of Staff
2022 • DesignOps Summit 2022
Gold
Adel Du Toit
Get Your CFO To Say: 'Our Strategic Goal is User Obsession'
2022 • Design at Scale 2022
Gold

More Videos

Alex Hurworth

"The health of our planet directly influences our health and survival."

Alex Hurworth Bonnie John Fahd Arshad Antoine Marin

Designing a Contact Tracing App for Universal Access

October 23, 2020

Laine Riley Prokay

"Mentors are wonderful partners for feedback, collaboration, and setting employees up for success."

Laine Riley Prokay Lisa Gordon

Carving a Path for Early Career DesignOps Practitioners

September 9, 2022

Eniola Oluwole

"TripAdvisor is really nine major business units each thinking like sub companies instead of one end-to-end user experience."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Nathan Shedroff

"Design researchers know customers better than almost anyone else in the organization, yet they are rarely invited into strategy processes."

Nathan Shedroff

Double Your Mileage: Use Your Research Strategically

March 31, 2020

Sam Proulx

"Mobile devices not only have built-in screen readers but also pitch-to-zoom and magnification settings out of the box."

Sam Proulx

Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World

November 17, 2022

Feleesha Sterling

"At LinkedIn, rapid research uncovered trends across products that helped reduce duplicate work."

Feleesha Sterling

Building a Rapid Research Program (Videoconference)

May 18, 2023

Neil Barrie

"Culturally resonant experiences can influence society in ways governments only dream of, with velocity and speed."

Neil Barrie

Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture

March 25, 2024

John Devanney

"For many years, folks came to us with questions beyond design — about career paths, tools, skills, and intake management."

John Devanney

The Design Management Office

November 6, 2017

Katy Mogal

"Surveys and logs tell us what users do, but qualitative tells us why."

Katy Mogal

But Do Your Insights Scale?

March 12, 2021