Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Let’s face it, many of us feel daunted by statistics. But we also know that colleagues and clients ask whether our research has “statistically significant” results. Erin’s book Design for Impact helps you to test your hypotheses about improving design, and she guides you through deciding on your effect sizes to help you get those statistically significant results. Caroline’s book Surveys That Work talks about “significance in practice” and she’s not all that convinced about whether it’s worth aiming for statistical significance. Watch this lively session where Erin and Caroline compared and contrasted their ideas and approaches - helped by your questions and contributions.
Key Insights
-
•
Understanding null hypothesis testing requires mentally flipping between positive and negative assumptions, which many find cognitively challenging.
-
•
The intimidating language and academic culture around statistics deter many practitioners from embracing it.
-
•
Moving fast is not the same as making progress; deliberate, slower decision-making aided by statistics leads to better outcomes.
-
•
Engineers are more receptive to statistics when involved early and can see clear benefits, though they resist extra coding work needed for experiments.
-
•
Statistical significance does not always equate to practical or meaningful significance in real-world decisions.
-
•
Effect size is a critical but often overlooked statistic that helps determine if a detected change is worth acting on.
-
•
Good survey design prioritizes asking useful questions over simply increasing the number of respondents.
-
•
Combining qualitative and quantitative research methods into a holistic approach improves understanding and reduces bias.
-
•
AI tools can provide rough drafts of survey questions but cannot replace human feedback tailored to specific audiences.
-
•
Building intuition about statistics and user data requires practice and embracing imperfection as part of learning.
Notable Quotes
"You have to think of a hypothesis, then the opposite null hypothesis, and try to disprove the null. It's quite confusing and flip-floppy."
"We're using math to try to understand the universe and what's real, and that makes statistics inherently hard."
"Fast is not a virtue in itself; we should move slower and act with intention to go in the right direction."
"Engineers don’t like writing more code, so extra work for experimentation can meet resistance unless they see clear learning outcomes."
"Statistical significance is a mathematical concept about chance; what decision makers often want is significance in practice."
"You get better results from asking one person a useful question than asking 10,000 people a silly question."
"Effect size tells you how large a change is, and many statistics books don’t cover it well enough."
"In research, hypothesis is just a fancy word for a guess you pull out of your ass to start testing."
"AI might give a first draft, but I still prefer asking real humans to understand how they interpret survey questions."
"Start with the basics: means, minimums, maximums, and ranges. It quickly becomes less mysterious and more fun."
Or choose a question:
More Videos
"Stakeholder interviews shouldn’t be the only thing democratized—there’s a whole breadth of methodologies to consider."
Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher NashDemocratization: Working with it, not against it [Advancing Research Community Workshop Series]
July 24, 2024
"You need sponsorship from senior leadership but also buy-in from all corners of the company to succeed."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"Sometimes clarity trumps consistency depending on the context or specific use case in the system."
Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn ResselDiscussion
June 9, 2016
"The word ‘key’ is an extra super duper important differentiator — among all priorities, these are the keys."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"It’s really hard to make changes to these systems because it’s risk-ridden and a major undertaking."
Malini RaoLessons Learned from a 4-year Product Re-platforming Journey
June 9, 2021
"85% of people were leaving after just looking at one page on the old site—that was a huge problem."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"If you want to talk about the things you’re learning, use the hashtag DAS 2021 on Twitter, Instagram, or LinkedIn."
Bria AlexanderOpening Remarks
June 9, 2021
"Growth boards enabled direction change sooner by objectively evaluating team health and strategic alignment."
Jackie HoLead Effectively While Preserving Team Autonomy with Growth Boards
January 8, 2024
"Soft eyes means zooming between the particular tree and the whole forest to see the bigger picture."
Dan HillDesigning for the infrastructures of everyday life
June 4, 2024