Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?

Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Peter Van Dijck
Coffee with Lou #4: Taking a Peek Under the Rosenbot's Hood
2024 • Rosenfeld Community
Samuel Proulx
From Standards to Innovation: Why Inclusive Design Wins
2025 • Advancing Service Design 2025
Gold
Barb Spanton
Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
2022 • Design at Scale 2022
Gold
Nicole Umphress
Delivering Design Education During a Global Pandemic: Lessons Learned
2022 • Design at Scale 2022
Gold
Craig Brookes
"Just Make it Look Good" and Other Ways We're Misunderstood
2021 • Design at Scale 2021
Gold
Sarah Gallimore
Inspire Progress with Artifacts from the Future
2022 • Civic Design 2022
Gold
Jim Kalbach
Peace is waged with sticky notes: Mapping Real-World Experiences
2018 • Enterprise Experience 2018
Gold
Changying (Z) Zheng
Navigating Innovation with Integrity
2024 • DesignOps Summit 2024
Gold
Steve Portigal
War Stories LIVE! Q&A-Discussion
2020 • Advancing Research 2020
Gold
Yasmine Khan
Checking Bias and Listening to Financially Vulnerable Americans
2020 • Advancing Research 2020
Gold
Jose Coronado
From Zero to Hero
2022 • DesignOps Summit 2022
Gold
Uday Gajendar
Day 2 Welcome
2024 • Designing with AI 2024
Gold
Caroline Jarrett
Garbage in, garbage out? Measuring error rates to get ready for AI
2026 • Rosenfeld Community
Courtney Kaplan
Taking it to the next level: Career paths in DesignOps
2018 • DesignOps Summit 2018
Gold
Dr. Jamika D. Burge
How UX researchers can partner with (and not be replaced by) AI [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Sarah Barrett
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
2025 • Rosenfeld Community

More Videos

Xenia Adjoubei

"We were interested in supporting networks of self-organization and mutual aid formed between refugees and host communities."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Tamara Kartoziia

"We are no longer just a collection of cars, we are a focused racing team ready to hit the track."

Tamara Kartoziia

Think global, adapt local: how service design accelerated B2B market entry by 6 months

November 20, 2025

Jay Bustamante

"AI tech debt has compounding interest to it."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Nicole Aleong

"Hope generates optimistic momentum towards a possibility becoming real."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Nick Lewis

"About 70 sites have been removed from the Low Carbon Showcase after regressing in sustainability."

Nick Lewis

Designing and building low-carbon websites independently

November 18, 2025

Vicky Teinaki

"Rather than inscrutable KPIs, we should find metrics that work in harmony with the needs of the user."

Vicky Teinaki Michele Marut Tim Parmee

Short Take #3: UX/Product Lessons from Your Industry Peers

December 6, 2022

Laura Weiss

"The true enabler of a successful innovation is not the new technology or clever business model, but the human beings involved."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Joerg Beringer

"We model context of use as knowledge graphs with tasks, sub-task goals, task objects—all linked in relations."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

June 10, 2025

Shelby Switzer

"Why be here if you’re not going to be here."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021