Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Have fun with statistics?
Thursday, December 12, 2024 • Rosenfeld Community
Share the love for this talk
Have fun with statistics?
Speakers: Caroline Jarrett and Erin Weigel
Link:

Summary

Let’s face it, many of us feel daunted by statistics. But we also know that colleagues and clients ask whether our research has “statistically significant” results. Erin’s book Design for Impact helps you to test your hypotheses about improving design, and she guides you through deciding on your effect sizes to help you get those statistically significant results. Caroline’s book Surveys That Work talks about “significance in practice” and she’s not all that convinced about whether it’s worth aiming for statistical significance. Watch this lively session where Erin and Caroline compared and contrasted their ideas and approaches - helped by your questions and contributions.

Key Insights

  • Understanding null hypothesis testing requires mentally flipping between positive and negative assumptions, which many find cognitively challenging.

  • The intimidating language and academic culture around statistics deter many practitioners from embracing it.

  • Moving fast is not the same as making progress; deliberate, slower decision-making aided by statistics leads to better outcomes.

  • Engineers are more receptive to statistics when involved early and can see clear benefits, though they resist extra coding work needed for experiments.

  • Statistical significance does not always equate to practical or meaningful significance in real-world decisions.

  • Effect size is a critical but often overlooked statistic that helps determine if a detected change is worth acting on.

  • Good survey design prioritizes asking useful questions over simply increasing the number of respondents.

  • Combining qualitative and quantitative research methods into a holistic approach improves understanding and reduces bias.

  • AI tools can provide rough drafts of survey questions but cannot replace human feedback tailored to specific audiences.

  • Building intuition about statistics and user data requires practice and embracing imperfection as part of learning.

Notable Quotes

"You have to think of a hypothesis, then the opposite null hypothesis, and try to disprove the null. It's quite confusing and flip-floppy."

"We're using math to try to understand the universe and what's real, and that makes statistics inherently hard."

"Fast is not a virtue in itself; we should move slower and act with intention to go in the right direction."

"Engineers don’t like writing more code, so extra work for experimentation can meet resistance unless they see clear learning outcomes."

"Statistical significance is a mathematical concept about chance; what decision makers often want is significance in practice."

"You get better results from asking one person a useful question than asking 10,000 people a silly question."

"Effect size tells you how large a change is, and many statistics books don’t cover it well enough."

"In research, hypothesis is just a fancy word for a guess you pull out of your ass to start testing."

"AI might give a first draft, but I still prefer asking real humans to understand how they interpret survey questions."

"Start with the basics: means, minimums, maximums, and ranges. It quickly becomes less mysterious and more fun."

Ask the Rosenbot
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Jemma Ahmed
Theme 2 Intro
2024 • Enterprise Experience 2020
Gold
Alison Rand
Scaling Impact with Service Design (Videoconference)
2021 • DesignOps Community
Theresa Marwah
How Atlassian is Operationalizing Respect in Research (Videoconference)
2020 • Advancing Research Community
Tricia Wang
SCALE: Discussion
2018 • Enterprise Experience 2018
Gold
Sahibzada Mayed
Cultivating Design Ecologies of Care, Community, and Collaboration
2023 • DesignOps Summit 2023
Gold
Dean Broadley
Not Black Enough to be White
2024 • DesignOps Summit 2020
Gold
What DesignOps Can Learn From DevOps
2017 • DesignOps Summit 2017
Gold
Ari Zelmanow
Dark Side of Democratization (Videoconference)
2023 • Advancing Research Community
Adrian Howard
Sturgeon’s Biases
2024 • DesignOps 2024
Gold
Erin Weigel
Get Your Whole Team Testing to Design for Impact
2024 • Rosenfeld Community
Julie Baher
Culture Change—My Journey
2015 • Enterprise UX 2015
Gold
Jennifer Strickland
Adopting a "Design By" Method
2021 • Civic Design 2021
Gold
Lavy Kumar
Future of Work
2021 • Design at Scale 2021
Gold
Victor Udoewa
Theme One Intro
2023 • Advancing Research 2023
Gold
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold

More Videos

"You’re born on third base but think you hit a triple — that’s the enterprise software delusion."

Standardizing Product Merits for Leaders, Designers, and Everyone

June 15, 2018

Simon Wardley

"Very few people in business are familiar with situational awareness unless they served in the military."

Simon Wardley

Maps and Topographical Intelligence (Videoconference)

January 31, 2019

Sandra Camacho

"There’s a gap between intentions and impact; humility is needed to close it."

Sandra Camacho

Creating More Bias-Proof Designs

January 22, 2025

Darian Davis

"By observing Jeff’s concerns and asking questions, I was able to show empathy and focus on the problem rather than get defensive."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Fisayo Osilaja

"I challenge everyone to create a strategic AI integration plan that goes beyond everyday tasks to achieve career and company goals."

Fisayo Osilaja

[Demo] The AI edge: From researcher to strategist

June 4, 2024

Uday Gajendar

"Working with CEOs like Mark Templeton is like Dancing with the Stars — an interpretive dance of translating fuzzy ideas."

Uday Gajendar

The Wicked Craft of Enterprise UX

May 13, 2015

Davis Neable

"Experience vision is a powerful tool because it gives everyone a tangible goal to work towards."

Davis Neable Guy Segal

How to Drive a Design Project When you Don’t Have a Design Team

June 10, 2021

Eniola Oluwole

"TripAdvisor is really nine major business units each thinking like sub companies instead of one end-to-end user experience."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Aurobinda Pradhan

"Design leaders want predictable roadmaps and healthy teams, measurable against timelines and goals."

Aurobinda Pradhan Shashank Deshpande

Introduction to Collaborative DesignOps using Cubyts

September 9, 2022