Rosenverse
Are My Research Findings Actually Meaningful?

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?

Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Jen Cardello
Standardizing Product Merits for Leaders, Designers, and Everyone
2018 • Enterprise Experience 2018
Gold
Jennifer Kanyamibwa
Creating the Blueprint: Growing and Building Design Teams
2018 • DesignOps Summit 2018
Gold
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Enterprise UX 2023
Gold
Angelos Arnis
Navigating the Rapid Shifts in Tech's Turbulent Terrain
2023 • DesignOps Summit 2023
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • DesignOps Summit 2021
Gold
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold
Bria Alexander
Welcome
2022 • DesignOps Summit 2022
Gold
Marissa Cui
Climate Design Product Showcase
2024 • Climate UX Interest Group
Eric Shumake
An AMA on UX's Role in Healthcare
2026 • Rosenfeld Community
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Matt LeMay
You Don’t “Get” Anyone to Do Anything
2022 • Design in Product 2022
Gold
Maggie Dieringer
Creating Consistency Through Constant Change
2024 • DesignOps Summit 2020
Gold
Megan Kierstead
You Are a Badass at UX: Overcoming Imposter Syndrome
2021 • Advancing Research 2021
Gold
Amahra Spence
Designing for Liberation, Rehearsing Freedom
2022 • Civic Design 2022
Gold
Christian Bason
Innovating With People: Unleashing the Potential of Civic Design
2021 • Civic Design 2021
Gold
Roberta Dombrowski
Making Research a Team Sport
2022 • Advancing Research 2022
Gold

More Videos

Anna Avrekh

"The transition from hands-on research to managing is a big mind shift where outcomes become more intangible."

Anna Avrekh Dr. John Pagonis Klara Pelcl Sina Schreiber

Expert Panel: Leading in and with Research

March 10, 2022

Taylor Klassman

"In a remote environment, building authentic, deep relationships with stakeholders is a key skill for researchers."

Taylor Klassman

Shaping the Next Era of UX Research: Collaborative Forum

March 11, 2025

Dr. Jamika D. Burge

"If we believe we're the only champions of human experience, then we’re not doing the right job."

Dr. Jamika D. Burge Steve Portigal Alba Villamil Sam Ladner

The Future of Research: Bridging the Gaps

July 29, 2021

Marisa Bernstein

"Compensation is so important to thank people for their time, and you will definitely build your recruitment pool much faster if you offer it."

Marisa Bernstein

It Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing

December 9, 2021

Dan Mall

"Design systems teams are most effective when they focus on serving the components and processes needed at scale by many teams."

Dan Mall

“Ask Me Anything” with Dan Mall, Author of Upcoming Rosenfeld Title, Design that Scales

October 2, 2023

Chris Geison

"I feel like probably the luckiest person here. And I have COVID."

Chris Geison

Theme 1 Intro

March 9, 2022

Bruce Gillespie

"Choose plain, everyday words over jargon to meet your audience where they are."

Bruce Gillespie

Learning from journalism: Balancing impactful communication with compassionate storytelling

March 13, 2025

Nancy Douyon

"The nobility complex is when we in the West unintentionally create solutions without accounting for our own biases."

Nancy Douyon

We'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex

June 15, 2018

Liz Ebengo

"There’s never really conflicting futures—everyone wants to take care of the most vulnerable children; you just have to find the bridge between visions."

Liz Ebengo

The Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward

December 4, 2024