Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.
Key Insights
-
•
Booking.com runs thousands of AB tests with a failure rate of about 90%, meaning most work is discarded but the few wins compound into massive growth.
-
•
Technical details like page load time can negate even visually superior design improvements, as shown by Erin's image optimization story.
-
•
Translation errors in multilingual platforms can cause user confusion and negatively affect conversion, highlighting the need for careful localization.
-
•
Typography inconsistencies, such as mixing serif and sans-serif fonts, materially impact legibility and conversion rates.
-
•
System fonts improve legibility and page load speed simultaneously, making them a strong choice over fancy custom fonts for UX optimization.
-
•
AB testing platforms that allow granular data slicing, like AB Smartly, empower deeper analysis of user segments and experiment impact.
-
•
Qualitative research is critical to generating valid experiment ideas and interpreting why tests succeed or fail beyond raw data.
-
•
Experiment duration depends heavily on traffic volume, business risk, and the importance of the decision, ranging from one week to several months.
-
•
Guardrail metrics such as customer service calls and repeat bookings help detect negative side effects of experiments beyond conversion.
-
•
Retesting ideas over time is valuable as external conditions, technology, and user contexts evolve, turning previous failures into wins.
Notable Quotes
"I’m actually a really big failure because nine out of 10 tests that we ran failed to show positive impact."
"Most work that most people do there ends up being thrown away."
"Booking.com was averaging more than 2 million room nights booked every day, so the scale is massive."
"Growth is driven by the compound effect where good upon good upon good eventually builds up to fast growth."
"Sometimes people ignore prior test results and test again, and occasionally they find a positive result."
"Execution usually fails, not the concept itself; small things like wrong colors or page load speed sneak in friction."
"Page load time is one of the main killers of conversion."
"I replaced Times New Roman with Arial and it ended up being conclusively and wildly positive for conversions."
"If you don’t need to use fancy typefaces, just don’t, because they can impact page speed and lose conversions."
"Guardrail metrics prevent you from improving conversion at the short term but damaging loyalty or increasing customer service calls."
Or choose a question:
More Videos
"Stakeholder interviews shouldn’t be the only thing democratized—there’s a whole breadth of methodologies to consider."
Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher NashDemocratization: Working with it, not against it [Advancing Research Community Workshop Series]
July 24, 2024
"Having a design system is not the end goal; it enables radical cost cuts, reduces tech debt, and improves workflows."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"Sometimes clarity trumps consistency depending on the context or specific use case in the system."
Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn ResselDiscussion
June 9, 2016
"If we are working on something that does not ladder up to our KSPs, then we should not be working on it."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"Designing for change management is just as important as the product experience strategy."
Malini RaoLessons Learned from a 4-year Product Re-platforming Journey
June 9, 2021
"Bringing stakeholders to observe user sessions helps them see that user-centered design isn’t just a cult."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"Day one of Design at Scale is going to be about ensuring craft on a massive scale."
Bria AlexanderOpening Remarks
June 9, 2021
"Growth boards demonstrate you can have agile software development at enterprise scale without rigid processes or chaos."
Jackie HoLead Effectively While Preserving Team Autonomy with Growth Boards
January 8, 2024
"Private cars are parked 95% of the time, making them highly inefficient in urban environments."
Dan HillDesigning for the infrastructures of everyday life
June 4, 2024