Are My Research Findings Actually Meaningful?
Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"The transition from hands-on research to managing is a big mind shift where outcomes become more intangible."
Anna Avrekh Dr. John Pagonis Klara Pelcl Sina SchreiberExpert Panel: Leading in and with Research
March 10, 2022
"In a remote environment, building authentic, deep relationships with stakeholders is a key skill for researchers."
Taylor KlassmanShaping the Next Era of UX Research: Collaborative Forum
March 11, 2025
"If we believe we're the only champions of human experience, then we’re not doing the right job."
Dr. Jamika D. Burge Steve Portigal Alba Villamil Sam LadnerThe Future of Research: Bridging the Gaps
July 29, 2021
"Compensation is so important to thank people for their time, and you will definitely build your recruitment pool much faster if you offer it."
Marisa BernsteinIt Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing
December 9, 2021
"Design systems teams are most effective when they focus on serving the components and processes needed at scale by many teams."
Dan Mall“Ask Me Anything” with Dan Mall, Author of Upcoming Rosenfeld Title, Design that Scales
October 2, 2023
"I feel like probably the luckiest person here. And I have COVID."
Chris GeisonTheme 1 Intro
March 9, 2022
"Choose plain, everyday words over jargon to meet your audience where they are."
Bruce GillespieLearning from journalism: Balancing impactful communication with compassionate storytelling
March 13, 2025
"The nobility complex is when we in the West unintentionally create solutions without accounting for our own biases."
Nancy DouyonWe'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex
June 15, 2018
"There’s never really conflicting futures—everyone wants to take care of the most vulnerable children; you just have to find the bridge between visions."
Liz EbengoThe Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward
December 4, 2024
