Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be linked to clear business imperatives and KPIs to avoid meaningless data collection.
-
•
Statistical significance alone does not determine if a finding is meaningful to business strategy.
-
•
Sample size and data variability significantly impact whether differences in scores are statistically significant.
-
•
Qualitative research provides crucial context and meaning that pure quantitative methods often miss.
-
•
Competitive benchmarking should include industries outside your own to meet evolving customer experience expectations.
-
•
Significant changes parallel to industry trends may not represent meaningful progress for a company.
-
•
Driver modeling helps prioritize research efforts by identifying which perceptions most influence key outcomes like NPS or loyalty.
-
•
Frequent measurement without implementing changes dilutes insights and wastes resources.
-
•
Researchers should tailor findings to the priorities of different stakeholders to increase buy-in and understanding.
-
•
A mixed-methods approach combining qualitative and quantitative data yields the most actionable and trustworthy insights.
Notable Quotes
"Research is not easy to do and it’s not something we should waste time in; it needs to be well thought out."
"A number like an NPS score means different things to different people depending on their role."
"Sample size has a direct impact on whether a change in score is statistically significant or just noise."
"Just because a change is statistically significant doesn’t mean it’s meaningful enough to change business strategy."
"When the pandemic hit, loyalty was shattered across industries; companies had to earn it back."
"If you measure without making changes, you shouldn’t be measuring more often — that’s a big no-no."
"If you had an infinite sample size, every change would be significant, which defeats the purpose."
"Driver modeling tells you which initiatives matter most so you know where to focus your research efforts."
"Attach your findings to what executives care about — increasing revenue, decreasing costs, innovating faster."
"I always do mixed methods because one alone doesn’t give you everything to truly understand what’s going on."
Or choose a question:
More Videos
"A kickback is a free flowing and life giving gathering that transforms and electrifies throughout the night."
Zariah CameronReDesigning Wellbeing for Equitable Care in the Workplace
September 23, 2024
"The real trick is being able to say no gracefully and communicate your priorities effectively."
John Cutler Harry MaxPrioritization for designers and product managers (1st of 3 seminars) (Videoconference)
June 13, 2024
"A connection that I made when I was 14 years old helped me to fund my college education, find my wife, and get the job at ExxonMobil that I still have today."
Nick CochranGrowing in Enterprise Design through Making Connections
June 3, 2019
"Not everything that is faced can be changed, but nothing can be changed until it is faced. - James Baldwin"
Deanna SmithLeading Change with Confidence: Strategies for Optimizing Your Process
September 23, 2024
"We’re now looking to help legislators think about digital outcomes as they draft new laws and regulations."
Magdalena ZadaraZero Hour: How to Get Far Quickly When Starting Your Digital Service Unit Late
November 16, 2022
"What gets us into trouble is not what we don’t know. It’s what we know for sure that just ain’t so."
Nova Wehman-BrownWe've Never Done This Before
June 4, 2019
"Machines may identify that higher ice cream sales is associated with higher band ice, but the former is not causing the latter."
Liwei DaiThe Heart and Brain of the AI Research
March 31, 2020
"The DDN shortened the network distance between any two government staff who found themselves thrown into COVID response work, making coordination faster."
Gordon Ross12 Months of COVID-19 Design and Digital Response with the British Columbia Government
December 8, 2021
"People will forget what you said, people will forget what you did, but people will never forget how you made them feel."
Tutti TaygerlyMake Space to Lead
June 12, 2021
