Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.
Key Insights
-
•
Booking.com runs thousands of AB tests with a failure rate of about 90%, meaning most work is discarded but the few wins compound into massive growth.
-
•
Technical details like page load time can negate even visually superior design improvements, as shown by Erin's image optimization story.
-
•
Translation errors in multilingual platforms can cause user confusion and negatively affect conversion, highlighting the need for careful localization.
-
•
Typography inconsistencies, such as mixing serif and sans-serif fonts, materially impact legibility and conversion rates.
-
•
System fonts improve legibility and page load speed simultaneously, making them a strong choice over fancy custom fonts for UX optimization.
-
•
AB testing platforms that allow granular data slicing, like AB Smartly, empower deeper analysis of user segments and experiment impact.
-
•
Qualitative research is critical to generating valid experiment ideas and interpreting why tests succeed or fail beyond raw data.
-
•
Experiment duration depends heavily on traffic volume, business risk, and the importance of the decision, ranging from one week to several months.
-
•
Guardrail metrics such as customer service calls and repeat bookings help detect negative side effects of experiments beyond conversion.
-
•
Retesting ideas over time is valuable as external conditions, technology, and user contexts evolve, turning previous failures into wins.
Notable Quotes
"I’m actually a really big failure because nine out of 10 tests that we ran failed to show positive impact."
"Most work that most people do there ends up being thrown away."
"Booking.com was averaging more than 2 million room nights booked every day, so the scale is massive."
"Growth is driven by the compound effect where good upon good upon good eventually builds up to fast growth."
"Sometimes people ignore prior test results and test again, and occasionally they find a positive result."
"Execution usually fails, not the concept itself; small things like wrong colors or page load speed sneak in friction."
"Page load time is one of the main killers of conversion."
"I replaced Times New Roman with Arial and it ended up being conclusively and wildly positive for conversions."
"If you don’t need to use fancy typefaces, just don’t, because they can impact page speed and lose conversions."
"Guardrail metrics prevent you from improving conversion at the short term but damaging loyalty or increasing customer service calls."
Dig deeper—ask the Rosenbot:
More Videos
"Survey question volume can range from one question to up to 40, but most participants answer around 20 questions."
Wyatt HaymanGlobal Research Panels (Videoconference)
August 8, 2020
"We’re far from done; evolving foundational elements, accessibility, and customization tools will drive Spectrum’s future."
PJ Buddhari Nate BaldwinMeet Spectrum, Adobe’s Design System
June 9, 2021
"If we center design on people and place before technology, we create more meaningful futures."
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"We don’t get upset when users say one thing and do another, but we freak out when our leadership behaves that way."
Peter MerholzThe Trials and Tribulations of Directors of UX (Videoconference)
July 13, 2023
"Hiring local community members as researchers in crisis zones creates safer environments and authentic stories."
Dr. Jamika D. Burge Mansi GuptaAdvancing the Inclusion of Womxn in Research Practices (Videoconference)
September 15, 2022
"You have to understand how your company actually makes money."
Amy MarquezINVEST: Discussion
June 15, 2018
"Telemetry is like sex in high school — everyone claims to do it, but most are doing it poorly."
Dane DeSutter Natalie Gedeon Deborah Hendersen Cheryl PlatzBeyond the Console: The rise of the Gamer Experience and how gaming will impact UX Research across industries (Videoconference)
May 17, 2024
"Jobs always ask for three references. I think I might start asking for three happy employees."
Zariah CameronReDesigning Wellbeing for Equitable Care in the Workplace
September 23, 2024
"Ask what helps you work better—this simple question is one of the most important things for colleagues and managers."
Jessica NorrisADHD: A DesignOps Superpower
September 9, 2022
