Are My Research Findings Actually Meaningful?
Summary
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"I jump out of bed every morning because an executive said for the first time, this feels like home, this is how I want to work."
Erin HauberDesign is Not the Frosting on the Scaled Agile Layer Cake
October 24, 2019
"Thoughts cause feelings, and feelings motivate actions. Changing thoughts changes everything."
Megan KiersteadYou Are a Badass at UX: Overcoming Imposter Syndrome
March 10, 2021
"Working at Uber requires grit."
Maggie DieringerCreating Consistency Through Constant Change
January 8, 2024
"Avoid too much upfront discovery that delays value and traps you into a fixed plan."
Meaghan Waters Fotina KoutropoulousLack of Product Thinking will Doom Your Legacy Modernization
June 9, 2021
"Market research can no longer afford to take six months with multiple waves; it has to be speedier and more agile now."
Jemma AhmedBringing together market and user research
October 17, 2019
"Start small by delivering the minimum amount of useful information, like a UX glossary or a single community of practice."
Maria TaylorKnowledge is Power: Managing the Lifeblood of the Design Org
October 3, 2023
"Accessibility is now considered not just a moral necessity and a human right but a business good because accessible products are better and easier for everyone to use."
Sam ProulxUnderstanding Screen Readers on Mobile: How And Why to Learn from Native Users
June 6, 2023
"We were spending 90% of our resource responding to tactical queries or performance reporting — essentially marking everyone's homework."
Catherine BlizzardUsing Integrated Insight to Drive Growth
March 10, 2022
"I know who you are and I have opinions."
Randolph Duke IIWar Stories LIVE! Randy Duke II
March 30, 2020
