Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
UX Lessons from running more than 1,200 A/B Tests
Summary
Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.
Key Insights
-
•
Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.
-
•
Execution often fails on good ideas rather than the ideas themselves.
-
•
Technical details like page load time can negate positive design changes if not carefully managed.
-
•
Edge case bugs in different languages or currencies can silently kill conversions.
-
•
Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.
-
•
Using system fonts improves legibility and page load speed, positively affecting conversions.
-
•
Tracking should only include users exposed to the tested change to avoid noisy data.
-
•
Qualitative user research drives many experiment ideas by revealing real user challenges.
-
•
Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.
-
•
Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.
Notable Quotes
"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."
"Compound effect means good upon good upon good eventually builds to incredibly fast growth."
"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."
"Increasing page load time by three seconds can nullify all your design improvements."
"Picking the wrong size image can impact customer experience as much as what you see on screen."
"There are as many versions of the website as stars in the sky because of language, currency, and device variations."
"No tracking is perfect because numbers tell you what likely happened, but not why it happened."
"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."
"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."
"You can’t fail 90 times out of 10 without laughing at yourself to keep going."
Or choose a question:
More Videos
"Employees can take as much or as little time off as they want, so long as their job gets done."
Lavy Kumar Kat Temple Shan Sebastian Tara Jensen Jenn ChouFuture of Work
June 9, 2021
"I’m most excited to get back in the office because I miss that culture and feeling like I’m working with friends every day."
Andreas Huebner Amy Takata Craig BrookesWhat Is It Like To Be Part of The UX Team at Compass?
March 11, 2021
"Perception maps visually represent how consumers perceive brands relative to competitors along key attributes."
Ricardo MartinsUnlocking the power of advanced quantitative methods
March 12, 2025
"Everybody involved in product making should have empathy for each other, not just for the customer."
Monty HammontreeThe Future of UX Research
December 3, 2020
"For me, ideally one design ops manager per 14 designers is the ratio that works best."
Benjamin RealShowing the Value of DesignOps by Not Having a DesignOps Team
October 21, 2020
"Some people have been doing research ops a lot longer than the term has existed."
Kate TowseyAsk Me Anything (AMA) with Kate Towsey
April 2, 2025
"The core of it is really putting yourself in that other person’s shoes—it’s basic but essential empathy."
Steve Portigal Chris Chapo Kelly Goto Christian RohrerDiscussion
May 13, 2015
"The more payment providers you offer, the higher the chance everyone can pick what works best for them."
Sam ProulxOnline Shopping: Designing an Accessible Experience
March 28, 2023
"Consistency between control labels and prototype instructions matters a lot, because mismatches can cause big problems for alternative navigation."
Sam ProulxPrototype Reviews, People With Disabilities, and You
October 1, 2021
Latest Books All books
Dig deeper with the Rosenbot
What is the ideal number of key insights to include in a UX research report?
How can I use an AI chatbot like Rosenbot to get quick answers on UX and product design topics?
How have changes in communication technologies historically affected collective knowledge building, and what might AI's role be?