Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be linked to clear business imperatives and KPIs to avoid meaningless data collection.
-
•
Statistical significance alone does not determine if a finding is meaningful to business strategy.
-
•
Sample size and data variability significantly impact whether differences in scores are statistically significant.
-
•
Qualitative research provides crucial context and meaning that pure quantitative methods often miss.
-
•
Competitive benchmarking should include industries outside your own to meet evolving customer experience expectations.
-
•
Significant changes parallel to industry trends may not represent meaningful progress for a company.
-
•
Driver modeling helps prioritize research efforts by identifying which perceptions most influence key outcomes like NPS or loyalty.
-
•
Frequent measurement without implementing changes dilutes insights and wastes resources.
-
•
Researchers should tailor findings to the priorities of different stakeholders to increase buy-in and understanding.
-
•
A mixed-methods approach combining qualitative and quantitative data yields the most actionable and trustworthy insights.
Notable Quotes
"Research is not easy to do and it’s not something we should waste time in; it needs to be well thought out."
"A number like an NPS score means different things to different people depending on their role."
"Sample size has a direct impact on whether a change in score is statistically significant or just noise."
"Just because a change is statistically significant doesn’t mean it’s meaningful enough to change business strategy."
"When the pandemic hit, loyalty was shattered across industries; companies had to earn it back."
"If you measure without making changes, you shouldn’t be measuring more often — that’s a big no-no."
"If you had an infinite sample size, every change would be significant, which defeats the purpose."
"Driver modeling tells you which initiatives matter most so you know where to focus your research efforts."
"Attach your findings to what executives care about — increasing revenue, decreasing costs, innovating faster."
"I always do mixed methods because one alone doesn’t give you everything to truly understand what’s going on."
Or choose a question:
More Videos
"The important thing was the conversation around the visual maps, not the size or scale of areas."
Shaping design, designers and teams
November 8, 2018
"Bringing design on an equal eye level with technology and business is the starting point."
Sabrina Mach Nina WainwrightHow to Design Your Design Operating Model
September 29, 2021
"If you think IE identity verification is necessary, ask why and if you really have to do it."
Samuel ProulxInvisible barriers: Why accessible service design can’t be an afterthought
December 3, 2024
"Accessibility is not just a checklist item that needs to be checked before something ships, it’s a methodology embedded from the start."
Saara Kamppari-MillerDesignOps for Inclusive Design and Accessibility (Videoconference)
May 26, 2022
"Product design and development can often mirror what happens in a chef’s kitchen: lots of coordination and spinning plates behind the scenes."
Briana ThomasThe Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
September 24, 2024
"A good story that people can retell is key for global communication, even if it distorts a bit."
Adam Cutler Karen Pascoe Ian Swinson Susan WorthmanDiscussion
June 8, 2016
"We had no idea where we were going, so we had to build and connect with teams like we’d never had before."
Operationalizing DesignOps
November 7, 2018
"Community is critical; solo practitioners must seek or create it to thrive and avoid isolation."
Leah Buley Joe NatoliAsk Me Anything with Leah Buley and Joe Natoli, co-authors of The User Experience Team of One (2nd edition)
October 8, 2024
"You need to learn the vocabulary, tools, and processes your peers know and understand where they are inadequate."
Nathan ShedroffDouble Your Mileage: Use Your Research Strategically
March 31, 2020