Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?
Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be linked to clear business imperatives and KPIs to avoid meaningless data collection.

  • Statistical significance alone does not determine if a finding is meaningful to business strategy.

  • Sample size and data variability significantly impact whether differences in scores are statistically significant.

  • Qualitative research provides crucial context and meaning that pure quantitative methods often miss.

  • Competitive benchmarking should include industries outside your own to meet evolving customer experience expectations.

  • Significant changes parallel to industry trends may not represent meaningful progress for a company.

  • Driver modeling helps prioritize research efforts by identifying which perceptions most influence key outcomes like NPS or loyalty.

  • Frequent measurement without implementing changes dilutes insights and wastes resources.

  • Researchers should tailor findings to the priorities of different stakeholders to increase buy-in and understanding.

  • A mixed-methods approach combining qualitative and quantitative data yields the most actionable and trustworthy insights.

Notable Quotes

"Research is not easy to do and it’s not something we should waste time in; it needs to be well thought out."

"A number like an NPS score means different things to different people depending on their role."

"Sample size has a direct impact on whether a change in score is statistically significant or just noise."

"Just because a change is statistically significant doesn’t mean it’s meaningful enough to change business strategy."

"When the pandemic hit, loyalty was shattered across industries; companies had to earn it back."

"If you measure without making changes, you shouldn’t be measuring more often — that’s a big no-no."

"If you had an infinite sample size, every change would be significant, which defeats the purpose."

"Driver modeling tells you which initiatives matter most so you know where to focus your research efforts."

"Attach your findings to what executives care about — increasing revenue, decreasing costs, innovating faster."

"I always do mixed methods because one alone doesn’t give you everything to truly understand what’s going on."

Ask the Rosenbot
Patrizia Bertini
DesignOps + KPIs = Measure your Impact!
2024 • DesignOps Summit 2020
Gold
Frances Yllana
DesignOps–Leading the Path to Parity (Videoconference)
2023 • DesignOps Community
Kaitlin Tasker
Fast and Fearless Inclusive Research
2023 • Advancing Research 2023
Gold
Tristin Oldani
Turning awareness into action with Climate UX
2025 • Climate UX Interest Group (Rosenfeld Community)
Prerna Makanawala
Achieving Balanced Design Consistency
2021 • Design at Scale 2021
Gold
Yoel Sumitro
Actions and Reflections: Bridging the Skills Gap among Researchers
2022 • Advancing Research 2022
Gold
Greg Petroff
Design is the Differentiator: Bringing New Design Innovations to a Very Antiquated and Very Large Industry
2021 • Design at Scale 2021
Gold
Lija Hogan
Three Years Out: Perspectives on the Near-Term Future of User Research (Videoconference)
2024 • Advancing Research Community
Erin Weigel
Real-world lessons to improve your conversion rates (Videoconference)
2024 • Rosenfeld Community
Jess Greco
Creating a Basis for Change: Scaling Design Maturity
2022 • Design at Scale 2022
Gold
Julie Baher
Culture Change—My Journey
2015 • Enterprise UX 2015
Gold
Jamie Beck Alexander
How can you find your role in climate? (Videoconference)
2024 • Climate UX Interest Group (Rosenfeld Community)
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Advancing Research 2022
Gold
Alison Rand
Scaling Impact with Service Design (Videoconference)
2021 • DesignOps Community
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Smitha Papolu
Theme 3 Discussion
2024 • Enterprise Experience 2020
Gold

More Videos

"Users’ perception predicts attrition and paid referrals — design absolutely matters when people decide to buy or go."

Standardizing Product Merits for Leaders, Designers, and Everyone

June 15, 2018

Simon Wardley

"The Red Queen effect means if others adapt, you must adapt or lose."

Simon Wardley

Maps and Topographical Intelligence (Videoconference)

January 31, 2019

Sandra Camacho

"The Reflexive Compass helps us discern bias patterns early, take accountability, and measure impact."

Sandra Camacho

Creating More Bias-Proof Designs

January 22, 2025

Darian Davis

"We’re all capable of creating and perpetuating toxic work relationships."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Fisayo Osilaja

"I challenge everyone to create a strategic AI integration plan that goes beyond everyday tasks to achieve career and company goals."

Fisayo Osilaja

[Demo] The AI edge: From researcher to strategist

June 4, 2024

Uday Gajendar

"Nobody wants to buy or use a sloppy product, especially when enterprise users engage daily for hours."

Uday Gajendar

The Wicked Craft of Enterprise UX

May 13, 2015

Davis Neable

"There is no design team responsible for the overarching experience across the entire company."

Davis Neable Guy Segal

How to Drive a Design Project When you Don’t Have a Design Team

June 10, 2021

Eniola Oluwole

"We stopped talking about patterns and consistency and started talking about scalability and speed to connect with stakeholders."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Aurobinda Pradhan

"Design producers love their creative tools and want to keep using them without multiple manual reporting steps."

Aurobinda Pradhan Shashank Deshpande

Introduction to Collaborative DesignOps using Cubyts

September 9, 2022