Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Have fun with statistics?

Thursday, December 12, 2024 • Rosenfeld Community
Share the love for this talk
Have fun with statistics?
Speakers: Caroline Jarrett and Erin Weigel
Link:

Summary

Let’s face it, many of us feel daunted by statistics. But we also know that colleagues and clients ask whether our research has “statistically significant” results. Erin’s book Design for Impact helps you to test your hypotheses about improving design, and she guides you through deciding on your effect sizes to help you get those statistically significant results. Caroline’s book Surveys That Work talks about “significance in practice” and she’s not all that convinced about whether it’s worth aiming for statistical significance. Watch this lively session where Erin and Caroline compared and contrasted their ideas and approaches - helped by your questions and contributions.

Key Insights

  • Statistical significance often confuses practitioners because it requires mentally flipping hypotheses and disproving nulls, which is cognitively demanding.

  • Effect size is critical to understanding whether a change detected by statistics is meaningful in practice, a concept often neglected in statistics education.

  • Fast progress isn't necessarily good progress; teams benefit from slowing down and using statistics to ensure they're moving in the right direction.

  • Engineers can be reluctant to implement experiments due to the extra coding load, but they respond well when they understand the learning value gained.

  • Survey results (the numerical outcomes) are often confused with the number of respondents required for statistical significance, leading to misunderstandings.

  • Statistical thresholds like 95% confidence can be adjusted depending on project needs; lower confidence levels are sometimes acceptable.

  • A good hypothesis often starts as an intuitive guess, which gets refined over time through repeated testing and data collection.

  • Qualitative and quantitative research should be viewed as complementary tools in a holistic research approach rather than opposed methods.

  • AI tools can help generate first drafts of survey questions, but human-centered pilot testing is essential to avoid errors and misinterpretations.

  • It's common and acceptable to act on results that are significant in practice but not statistically significant, especially when outcomes clearly affect users.

Notable Quotes

"Statistics is hard because you have to flip flop in your head: think of a hypothesis, then a null hypothesis, then try to disprove the null."

"People confuse statistical significance with significance in practice — they want to know if the change is meaningful, not just mathematically significant."

"Fast is not a virtue in and of itself; moving slower and acting with intention ensures you go in the right direction."

"Engineers hate writing more code, so getting them to buy into experiments means showing the value of the learning on the other side."

"You can have an effect size that matters in practice but isn’t statistically significant, like five users failing a key task in usability testing."

"Most science starts with somebody pulling a number out of their ass — it’s okay to start with a gut instinct or guess."

"Statistics is another tool in our toolbox, part of a hierarchy of evidence that includes qualitative and quantitative methods."

"AI can create first drafts of survey questions, but unless you pilot test with real humans, you won’t know if your audience gets it."

"A lot of people think 95% confidence is the only way, but you can adjust confidence levels based on your situation and needs."

"Start with basics like means, minimums, and ranges — statistics rapidly becomes less mysterious and more useful with practice."

Ask the Rosenbot
Cara Maritz
The Art of Extrapolation
2023 • Advancing Research 2023
Gold
Jorge Arango
Design as an Antidote to VUCA
2019 • Enterprise Community
Wendy Johansson
Design at Scale: Behind the Scenes
2021 • Enterprise Community
Bria Alexander
Opening Remarks Day 1
2024 • Advancing Research 2024
Gold
Trisha Causley
[Demo] Complexity in disguise: Crafting experiences for generative AI features
2024 • Designing with AI 2024
Gold
Julie Gitlin
Design as an Agent of Digital Transformation at JPMC
2021 • Design at Scale 2021
Gold
Daniel J. Rosenberg
Designing with and for Artificial Intelligence
2022 • Enterprise Community
Margot Bloomstein
Fostering Trust in Your Brand and Beyond
2020 • Enterprise Community
Sam Proulx
Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
2022 • Civic Design 2022
Gold
Nalini P. Kotamraju
An Organizational Story: Salesforce Lightning Design System
2016 • Enterprise UX 2016
Gold
Ruzanna Rozman
Getting in Flow with Your Team
2024 • DesignOps Summit 2020
Gold
Ali Jeffery
How DesignOps Helped Enable Wall Street to Work Remotely
2020 • DesignOps Summit 2020
Gold
Daniel Orbach
Zero to One: Co-Creating Operating Models with your Team
2024 • DesignOps Summit 2024
Gold
Chloe Amos-Edkins
A Cultural Approach: Research in the Context of Glocalisation
2023 • Advancing Research 2023
Gold
Kate Towsey
Ask Me Anything (AMA) with Kate Towsey
2025 • Rosenfeld Community
Josina Vink
Navigating the pitfalls of systems thinking in service design
2024 • Advancing Service Design 2024
Gold

More Videos

John Cutler

"You have to go from being able to deliver a message in three bullets and then turn around in the next moment and deliver a massive service blueprint or customer journey."

John Cutler

Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)

December 3, 2024

Darian Davis

"We’re all capable of creating and perpetuating toxic work relationships."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Dave Gray

"If you’re having more conversation about what the actual question means than answering it, that’s a problem with the canvas we need to fix."

Dave Gray

Group Activity: Making Sense of DesignOps

November 7, 2017

John Cutler

"If you are a glue person expecting to be indispensable, you are more dispensable than you think. But if you’re a catalyst moving information and translating to action, that’s more valuable now."

John Cutler

The Alignment Trap

November 29, 2023

Abby Covert

"Providing alternative content is often a kinder gesture than any lengthy alternative text could be."

Abby Covert

Stuck? Diagrams Help

October 27, 2022

Mark Interrante

"Use four simple questions to find out what people really want and why — it avoids misaligned efforts."

Mark Interrante

Collaboration Flows in Product Development

June 9, 2017

Devon Powers

"We can’t just think technology or innovation will solve our problems; we have to embrace and work with those problems."

Devon Powers

Imagining Better Futures

March 9, 2022

Prayag Narula

"Your primary function as a research team is to build a culture of research, not just react to project requests."

Prayag Narula

How to Empower Your Designers to Do Good Research – And Why You Want To

June 10, 2022

Peter Van Dijck

"Your short term memory is the context window; too big, and the model has trouble accessing everything."

Peter Van Dijck

Hands on AI #3: Claude Code for UX people

October 22, 2025