Experimental research: techniques for deep, psychology-driven insights
Summary
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"We hadn’t really thought about governance up front at all, which is not the ideal way to do something like that."
Dave Malouf Amy ThibodeauPanel: Design Systems and Documentation
November 7, 2017
"I thought, maybe now is the time for change, maybe designers should code—maybe the better question in 2025 is should I code."
John DonmoyerShipping your code generation experiments to production
June 11, 2025
"It’s critical to measure perceived value and ask: Are you really solving the right problem with AI?"
Clara Kliman-SilverUX Futures: The Role of Artificial Intelligence in Design
June 7, 2023
"If you want to receive value, you have to create value first; expecting it before contributing is a mistake."
Mark TempletonCreating a Legacy: the ultimate experience
June 9, 2017
"The mayor wanted to shrink the neighborly bonds that the city was known for by investing in a 2030 living Monument of collaboration and community."
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"Fixing problems found for one disability group often helps fix problems for others too."
Sam ProulxPrototype Reviews, People With Disabilities, and You
December 8, 2021
"I thought it was really important to start closing the gap by collating monthly themes and sharing them with the wider team."
Anna Nguyen Emily BroganWhy Our Voice of the Customer is Better Than Yours
March 10, 2022
"Maybe the person who is the loudest in the room isn’t the one you should manage closely as a stakeholder."
Anna Poznyakov Richa PrajapatiGet The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact
March 12, 2021
"I don’t exactly even know who I am if not a researcher."
Nalini P. KotamrajuTwo Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"
March 10, 2021