Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.
Key Insights
-
•
Booking.com runs thousands of AB tests with a failure rate of about 90%, meaning most work is discarded but the few wins compound into massive growth.
-
•
Technical details like page load time can negate even visually superior design improvements, as shown by Erin's image optimization story.
-
•
Translation errors in multilingual platforms can cause user confusion and negatively affect conversion, highlighting the need for careful localization.
-
•
Typography inconsistencies, such as mixing serif and sans-serif fonts, materially impact legibility and conversion rates.
-
•
System fonts improve legibility and page load speed simultaneously, making them a strong choice over fancy custom fonts for UX optimization.
-
•
AB testing platforms that allow granular data slicing, like AB Smartly, empower deeper analysis of user segments and experiment impact.
-
•
Qualitative research is critical to generating valid experiment ideas and interpreting why tests succeed or fail beyond raw data.
-
•
Experiment duration depends heavily on traffic volume, business risk, and the importance of the decision, ranging from one week to several months.
-
•
Guardrail metrics such as customer service calls and repeat bookings help detect negative side effects of experiments beyond conversion.
-
•
Retesting ideas over time is valuable as external conditions, technology, and user contexts evolve, turning previous failures into wins.
Notable Quotes
"I’m actually a really big failure because nine out of 10 tests that we ran failed to show positive impact."
"Most work that most people do there ends up being thrown away."
"Booking.com was averaging more than 2 million room nights booked every day, so the scale is massive."
"Growth is driven by the compound effect where good upon good upon good eventually builds up to fast growth."
"Sometimes people ignore prior test results and test again, and occasionally they find a positive result."
"Execution usually fails, not the concept itself; small things like wrong colors or page load speed sneak in friction."
"Page load time is one of the main killers of conversion."
"I replaced Times New Roman with Arial and it ended up being conclusively and wildly positive for conversions."
"If you don’t need to use fancy typefaces, just don’t, because they can impact page speed and lose conversions."
"Guardrail metrics prevent you from improving conversion at the short term but damaging loyalty or increasing customer service calls."
Or choose a question:
More Videos
"When you use assistive technology, consistency isn’t just nice—it’s the difference between being able to buy and having to give up."
Sam ProulxOnline Shopping: Designing an Accessible Experience
June 7, 2023
"Designing procurement processes can fundamentally change outcomes and experiences inside and outside government."
Bria Alexander Ariel Kennan Charlotte Lee Sarah Brooks Emily Lessard Gordon Ross Joanne DongReflect and Chart Forward
December 10, 2021
"Provoking and reframing perspective helps teams think differently and get unstuck."
Gina MendoliaTherapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
December 3, 2024
"Current AI adoption is obsessed with efficiency and transactional tool features, which feels more engineering than experiential."
Josh Clark Veronika KindredSentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
January 15, 2025
"Individual contributors can create psychological safety by building intentional support networks."
Alla WeinbergPeople Are Sick of Change: Psychological Safety is the Cure (Videoconference)
July 20, 2023
"Badly designed personas create stereotypes, not archetypes."
Cassini Nazir Meah LinThe Dangers of Empathy: Toward More Responsible Design Research
March 27, 2023
"Design ops reporting separately from design managers creates a safe space—sometimes our DPMs are the therapists of the team."
Brennan HartichCommunicating and Establishing DesignOps as a New Function
November 7, 2018
"The trusted partnership between the Chief of Staff and the executive is critically important."
Isaac HeyveldExpand DesignOps Leadership as a Chief of Staff
September 8, 2022
"We lead ourselves, our peers, our stakeholders; leadership is not about title or level."
Etienne FangThe Power of Care: From Human-Centered Research to Humanity-Centered Leadership
March 10, 2021
