Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Russ Unger
Onboarding: The Ecosystem, not the Afterthought
2017 • DesignOps Summit 2017
Gold
William Newton
How to Lead With Data, and Without Data
2023 • Enterprise UX 2023
Gold
Bria Alexander
Day 3 Welcome
2024 • DesignOps Summit 2024
Gold
Sahibzada Mayed
The Politics of Radical Research: A Manifesto
2023 • Advancing Research 2023
Gold
Phil Gilbert
A Consistent Culture of Design
2015 • Enterprise UX 2015
Gold
Dharani Perera
The mandala of service design: unlocking alignment and action through service design
2025 • Advancing Service Design 2025
Gold
Allison Sanders
Operating with Purpose
2024 • DesignOps Summit 2020
Gold
Steve Portigal
War Stories LIVE! Q&A-Discussion
2020 • Advancing Research 2020
Gold
Alla Weinberg
Healing Toxic Stress
2024 • DesignOps Summit 2024
Gold
Sam Proulx
Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
2022 • DesignOps Summit 2022
Gold
James Lang
If you can design an app, you can design a community
2025 • Rosenfeld Community
Ned Dwyer
Right horses for the right courses – how and when to democratize research
2025 • Advancing Service Design 2025
Gold
Andy Barraclough
From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
2024 • DesignOps Summit 2024
Gold
Dave Hora
A Research Skills Evolution
2021 • Advancing Research 2021
Gold
Milan Guenther
The $212 billion ‘so what?’: unlocking impact in development cooperation
2025 • Advancing Service Design 2025
Gold

More Videos

Taylor Jennings

"We had to convince our director and executives by showing how the repository added real value for the product team."

Taylor Jennings Joe Nelson Alex Knoll

Repository Retrospective: Learnings from Introducing a Central Place for UX Research

March 9, 2022

Nicole Aleong

"Expectation describes how things ought to be and reinforces the distinction between the present and what is not yet here."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Jeff Gothelf

"Innovation labs don’t work because their mission is to be innovative, not necessarily to solve real customer problems."

Jeff Gothelf

Innovation Studios: the Engines of Enterprise Experimentation

May 14, 2015

Saara Kamppari-Miller

"Leadership needs to be just as behind accessibility as the designers and professionals doing the work."

Saara Kamppari-Miller

DesignOps for Inclusive Design and Accessibility

May 26, 2022

Tess Dixon

"We need to reject the notion that work is our only identity."

Tess Dixon

C'mon Get Happy

September 29, 2021

Liam Thurston

"If your team sees a clear path of mastery in your organization, they’re much more likely to stay."

Liam Thurston

Why Your Design Team Is Quitting, And How To Fix It

June 10, 2022

Ian Swinson

"It’s huge to have a framework so managers and employees speak the same language around skills and growth."

Ian Swinson

Designing and Driving UX Careers

June 8, 2016

Leisa Reichelt

"At Atlassian, many confident non-researchers were already doing lots of research, which was an opportunity and a challenge."

Leisa Reichelt

Opening Keynote: Operating in Context

November 7, 2018

Rachael Greene

"Consistent, gentle support over time creates deeply loyal adopters."

Rachael Greene Alison Davis

Building a Design Ops Practice that Really Works (Most of the Time)

October 2, 2025