Rosenverse
Experimental research: techniques for deep, psychology-driven insights

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Experimental research: techniques for deep, psychology-driven insights

Gold
Wednesday, March 12, 2025 • Advancing Research 2025
Share the love for this talk
Experimental research: techniques for deep, psychology-driven insights
Speakers: Katie Hansen
Link:

Summary

In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.

Key Insights

  • •

    Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.

  • •

    Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.

  • •

    Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.

  • •

    Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.

  • •

    Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.

  • •

    Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.

  • •

    Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.

  • •

    Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.

  • •

    Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.

  • •

    Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.

Notable Quotes

"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."

"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."

"Randomization ensures differences between groups come from the variable tested, not external factors."

"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."

"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."

"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."

"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."

"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."

"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."

"Start small with one clear testable hypothesis and build from there to make experimental research approachable."

Ask the Rosenbot
Steve Baty
Discussion
2016 • Enterprise UX 2016
Gold
Greg Petroff
Design is the Differentiator: Bringing New Design Innovations to a Very Antiquated and Very Large Industry
2021 • Design at Scale 2021
Gold
Caroline Vize
The State of UX: Five Lessons from 2021 to Accelerate Digital Experience in 2022
2022 • Advancing Research 2022
Gold
Ian Swinson
Designing and Driving UX Careers
2016 • Enterprise UX 2016
Gold
Dan Ward
Failure Friday #1 with Dan Ward
2025 • Rosenfeld Community
Jon Fukuda
Theme 3 Intro
2024 • DesignOps Summit 2024
Gold
Smitha Papolu
Theme 3 Discussion
2019 • Enterprise Experience 2019
Gold
Bria Alexander
Welcome
2024 • Enterprise Experience 2020
Gold
Anna Avrekh
Expert Panel: Leading in and with Research
2022 • Advancing Research 2022
Gold
Jacqui Frey
Setting the Table for Dynamic Change
2019 • DesignOps Summit 2019
Gold
Jeff Gothelf
The Intersection of Lean and Design
2019 • Enterprise Community
Jon Fukuda
All the Ops: Successful cross-functional collaboration
2025 • DesignOps Summit 2025
Gold
Josh Clark
Sentient Design: New Postures for AI-Mediated Experiences (2nd of 3 seminars)
2025 • Rosenfeld Community
Abbey Smalley
Scaling UX Past the Size of Your Team
2024 • Enterprise Experience 2020
Gold
Shipra Kayan
How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
2021 • DesignOps Summit 2021
Gold
Jessica Norris
ADHD: A DesignOps Superpower
2022 • DesignOps Summit 2022
Gold

More Videos

Dave Malouf

"We hadn’t really thought about governance up front at all, which is not the ideal way to do something like that."

Dave Malouf Amy Thibodeau

Panel: Design Systems and Documentation

November 7, 2017

John Donmoyer

"I thought, maybe now is the time for change, maybe designers should code—maybe the better question in 2025 is should I code."

John Donmoyer

Shipping your code generation experiments to production

June 11, 2025

Clara Kliman-Silver

"It’s critical to measure perceived value and ask: Are you really solving the right problem with AI?"

Clara Kliman-Silver

UX Futures: The Role of Artificial Intelligence in Design

June 7, 2023

Mark Templeton

"If you want to receive value, you have to create value first; expecting it before contributing is a mistake."

Mark Templeton

Creating a Legacy: the ultimate experience

June 9, 2017

Sarah Gallimore

"The mayor wanted to shrink the neighborly bonds that the city was known for by investing in a 2030 living Monument of collaboration and community."

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Sam Proulx

"Fixing problems found for one disability group often helps fix problems for others too."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

December 8, 2021

Anna Nguyen

"I thought it was really important to start closing the gap by collating monthly themes and sharing them with the wider team."

Anna Nguyen Emily Brogan

Why Our Voice of the Customer is Better Than Yours

March 10, 2022

Anna Poznyakov

"Maybe the person who is the loudest in the room isn’t the one you should manage closely as a stakeholder."

Anna Poznyakov Richa Prajapati

Get The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact

March 12, 2021

Nalini P. Kotamraju

"I don’t exactly even know who I am if not a researcher."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021