Are My Research Findings Actually Meaningful?
Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"We were interested in supporting networks of self-organization and mutual aid formed between refugees and host communities."
Xenia Adjoubei Sean BruceEmpowering Communities Through the Researcher in Residence Program
March 29, 2023
"We are no longer just a collection of cars, we are a focused racing team ready to hit the track."
Tamara KartoziiaThink global, adapt local: how service design accelerated B2B market entry by 6 months
November 20, 2025
"AI tech debt has compounding interest to it."
Jay BustamanteNavigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation
October 2, 2023
"Hope generates optimistic momentum towards a possibility becoming real."
Nicole AleongFuture Orientations to Everyday Life: Futures Anthropology as a Methodology
March 26, 2024
"About 70 sites have been removed from the Low Carbon Showcase after regressing in sustainability."
Nick LewisDesigning and building low-carbon websites independently
November 18, 2025
"Rather than inscrutable KPIs, we should find metrics that work in harmony with the needs of the user."
Vicky Teinaki Michele Marut Tim ParmeeShort Take #3: UX/Product Lessons from Your Industry Peers
December 6, 2022
"The true enabler of a successful innovation is not the new technology or clever business model, but the human beings involved."
Laura WeissTurn Down the Heat: 3 Ways to Handle Conflict in the Moment
November 20, 2024
"We model context of use as knowledge graphs with tasks, sub-task goals, task objects—all linked in relations."
Joerg Beringer Thomas GeisScaling User Research with AI: Continuous Discovery of User Needs in Minutes
June 10, 2025
"Why be here if you’re not going to be here."
Shelby SwitzerMaking Space for Community Knowledge-sharing in a Distributed World
December 10, 2021
