Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.
Key Insights
-
•
Booking.com runs thousands of AB tests with a failure rate of about 90%, meaning most work is discarded but the few wins compound into massive growth.
-
•
Technical details like page load time can negate even visually superior design improvements, as shown by Erin's image optimization story.
-
•
Translation errors in multilingual platforms can cause user confusion and negatively affect conversion, highlighting the need for careful localization.
-
•
Typography inconsistencies, such as mixing serif and sans-serif fonts, materially impact legibility and conversion rates.
-
•
System fonts improve legibility and page load speed simultaneously, making them a strong choice over fancy custom fonts for UX optimization.
-
•
AB testing platforms that allow granular data slicing, like AB Smartly, empower deeper analysis of user segments and experiment impact.
-
•
Qualitative research is critical to generating valid experiment ideas and interpreting why tests succeed or fail beyond raw data.
-
•
Experiment duration depends heavily on traffic volume, business risk, and the importance of the decision, ranging from one week to several months.
-
•
Guardrail metrics such as customer service calls and repeat bookings help detect negative side effects of experiments beyond conversion.
-
•
Retesting ideas over time is valuable as external conditions, technology, and user contexts evolve, turning previous failures into wins.
Notable Quotes
"I’m actually a really big failure because nine out of 10 tests that we ran failed to show positive impact."
"Most work that most people do there ends up being thrown away."
"Booking.com was averaging more than 2 million room nights booked every day, so the scale is massive."
"Growth is driven by the compound effect where good upon good upon good eventually builds up to fast growth."
"Sometimes people ignore prior test results and test again, and occasionally they find a positive result."
"Execution usually fails, not the concept itself; small things like wrong colors or page load speed sneak in friction."
"Page load time is one of the main killers of conversion."
"I replaced Times New Roman with Arial and it ended up being conclusively and wildly positive for conversions."
"If you don’t need to use fancy typefaces, just don’t, because they can impact page speed and lose conversions."
"Guardrail metrics prevent you from improving conversion at the short term but damaging loyalty or increasing customer service calls."
Or choose a question:
More Videos
"As a design leader, you must keep everyone moving in the same direction, despite complex tasks and high emotional stakes."
This Game is Never Done: Design Leadership Techniques from the Video Game World
November 6, 2017
"When times are uncertain, transparency helps tremendously."
Abby CovertStuck? Diagrams Help (Videoconference)
October 27, 2022
"The richness of mixed methodologies is often lost on those who only value metrics and consistency."
Steve Portigal Susan Simon-Daniels Tamara Hale Randolph Duke IIWar Stories LIVE! Q&A-Discussion
March 30, 2020
"We always tried to leave time in meetups for people to chat and get to know each other."
Kara KaneCommunities of Practice for Civic Design (Videoconference)
April 7, 2022
"UX research is a journey, not a simple flick of a switch – every challenge you face, someone else has been through."
Caroline VizeThe State of UX: Five Lessons from 2021 to Accelerate Digital Experience in 2022
March 9, 2022
"Sometimes growth is vertical, but we might take different paths and grow sideways and literally move."
Lada GorlenkoTheme 3: Introduction
June 10, 2021
"As a party planner, your role is to ensure all the right people are invited to the AI development process."
Jay BustamanteNavigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation
October 2, 2023
"I don’t want to lose my job by teaching designers how to do our job, but teaching one method felt safe."
Marjorie Stainback Kelsey KingmanTransforming Strategic Research Capacity through Democratization
October 24, 2019
"Trauma is a psychological experience shaped by individual perception, not just a traumatic event itself."
Rachael Dietkus, LCSW Uday Gajendar Dr. Dawn Emerick Dawn E. Shedrick, LCSWLeading through the long tail of trauma (Videoconference)
July 13, 2022