Rosenverse
Are My Research Findings Actually Meaningful?

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?

Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Nathan Shedroff
Redefining Value: Bridging the Innovation Culture Divide
2015 • Enterprise UX 2015
Gold
Matt Bernius
Trauma-informed Research: A Panel Discussion
2021 • Advancing Research Community
Mackenzie Guinon
M.C. Escher’s UX Research Career Ladder
2022 • Advancing Research 2022
Gold
Megan Blocker
Getting to the “So What?”: How Management Consulting Practices Can Transform Your Approach to Research
2024 • Advancing Research 2024
Gold
Jackie Ho
Lead Effectively While Preserving Team Autonomy with Growth Boards
2024 • Enterprise Experience 2020
Gold
James Chudley
Decarbonising User Journeys: How minimising enables us to do more with less
2025 • Climate UX Interest Group
Bria Alexander
State of DesignOps Panel
2021 • DesignOps Summit 2021
Gold
Megan Blocker
A Selectively Scrappy Approach to ResearchOps
2018 • DesignOps Summit 2018
Gold
Dave Hora
Research in the Face of Complexity: New Sensibility for New Situations
2025 • Rosenfeld Community
Kit Unger
Theme 1 Intro
2022 • Design at Scale 2022
Gold
Alfred Kahn
A Seat at the Table: Making Your Team a Strategic Partner
2023 • Design in Product 2023
Gold
David Conrad
The Feeling of Data
2023 • Enterprise Community
Gina Mendolia
Therapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
2024 • Advancing Service Design 2024
Gold
Matt Duignan
Atomizing Research: Trend or Trap
2020 • Advancing Research 2020
Gold
Saara Kamppari-Miller
Inclusive Design is DesignOps
2021 • DesignOps Summit 2021
Gold
Joanna Vodopivec
One Research Team for All - Influence Without Authority
2022 • Advancing Research 2022
Gold

More Videos

Carl Turner

"I found the problem solver in the UI developer and identified the project’s sponsor who really felt the pain."

Carl Turner

You Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project

March 28, 2023

Karen Pascoe

"Performance management is really about the employee experience; avoid surprises by coaching continuously on both what and how folks are doing."

Karen Pascoe

Developing Experience Teams and Talent in the Enterprise

June 8, 2016

Jilanna Wilson

"We share the inconvenience of meeting times across time zones, alternating so no one team always bears it."

Jilanna Wilson

Distributed Design Operations Management

October 23, 2019

Shawna Hein

"The collaboration cycle requires following standards for accessibility, content, design system usage, and IA or you don’t launch your product."

Shawna Hein Kevin Hoffman

Create a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts

November 17, 2022

Barb Spanton

"I still kind of pause in my tracks when I see our corporate vision: a world without fear of cancer."

Barb Spanton

Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'

June 10, 2022

Briana Thomas

"As the delivery driver, I connected the dots, got people briefs, and sometimes said no to lofty leadership requests."

Briana Thomas

The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams

September 24, 2024

Gina Mendolia

"Coaches organize practices or drills to push players toward specific goals and improvement."

Gina Mendolia

Therapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems

December 3, 2024

Lija Hogan

"Now everyone’s an edge case, so research needs to focus on those edge cases, not just the average user."

Lija Hogan Milan Mijatovic Sam Proulx Louis Rosenfeld

Three Years Out: Perspectives on the Near-Term Future of User Research

March 15, 2024

John Calhoun

"A design ops career ladder can look like a step stool unless you create multiple tracks for growth and specialization."

John Calhoun Rachel Posman

Two Sides of the DesignOps Coin: Teams Ops and Product Ops

January 8, 2024