Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Josh Clark
Sentient Design: New Design Patterns for New Experiences (3rd of 3 seminars)
2025 • Rosenfeld Community
Julie Baher
Culture Change—My Journey
2015 • Enterprise UX 2015
Gold
John Devanney
The Design Management Office
2017 • DesignOps Summit 2017
Gold
Rachael Dietkus, LCSW
Leading through the long tail of trauma
2022 • Enterprise Community
Ned Gartside
Navigating accessibility and climate
2024 • Climate UX Interest Group
Jemma Ahmed
Research at an inflection point: Adapting to a new era of collaboration, equity, and innovation
2025 • Advancing Research 2025
Gold
Failure Friday #4: Invisible Work: How I Stalled My Career by Not Showing My Work
2025 • Rosenfeld Community
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Design in Product 2023
Gold
Anat Fintzi
Delivering at Scale: Making Traction with Resistant Partners
2022 • Design at Scale 2022
Gold
Richard Buchanan
Creativity and Principles in the Flourishing Enterprise
2018 • Enterprise Experience 2018
Gold
Panel Discussion: Communicating the Value of DesignOps
2018 • DesignOps Summit 2018
Gold
Theresa Slate
Why Changing Hearts & Minds Doesn’t Work When Promoting DE&I Efforts, but Checklists Do
2023 • DesignOps Summit 2023
Gold
Robin Beers
Beyond Insights: Researchers as Organizational Change Catalysts
2024 • Advancing Research 2024
Gold
Jackie Ajoux
Leveling-Up: A Single-Player’s Guide to the DesignOps Team-of-One
2024 • DesignOps Summit 2020
Gold
JD Buckley
Communicating the ROI of UX within a large enterprise and out on the streets
2018 • Enterprise Experience 2018
Gold
Greg Petroff
Exit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian
2025 • Rosenfeld Community

More Videos

John Cutler

"Patient opportunism means most of the time having the current at your back and only occasionally angling directly against it to avoid burnout."

John Cutler

Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)

December 3, 2024

Darian Davis

"Reviewing decisions out loud held everyone accountable and made it harder for Jeff to forget or change his mind."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Dave Gray

"The tool is designed so you can just sit down, read the questions, and get started even if you don’t have much background knowledge."

Dave Gray

Group Activity: Making Sense of DesignOps

November 7, 2017

John Cutler

"Executives only like things in threes, but if you do a three by three, you can get away with nine boxes—that's the max executive information processing."

John Cutler

The Alignment Trap

November 29, 2023

Abby Covert

"Just because you can color code something doesn’t mean you should."

Abby Covert

Stuck? Diagrams Help

October 27, 2022

Mark Interrante

"You can see our organization through the homepage of our site — we each owned a rectangle."

Mark Interrante

Collaboration Flows in Product Development

June 9, 2017

Devon Powers

"Trend forecasting is a culture industry—it produces culture as much as it predicts it."

Devon Powers

Imagining Better Futures

March 9, 2022

Prayag Narula

"Unbiased qualitative research does not exist. The best we can do is acknowledge and minimize biases."

Prayag Narula

How to Empower Your Designers to Do Good Research – And Why You Want To

June 10, 2022

Peter Van Dijck

"Skills are like onboarding material for the model; you write them as simple markdown files that describe workflows or tools."

Peter Van Dijck

Hands on AI #3: Claude Code for UX people

October 22, 2025