Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.
Key Insights
-
•
Booking.com runs thousands of AB tests with a failure rate of about 90%, meaning most work is discarded but the few wins compound into massive growth.
-
•
Technical details like page load time can negate even visually superior design improvements, as shown by Erin's image optimization story.
-
•
Translation errors in multilingual platforms can cause user confusion and negatively affect conversion, highlighting the need for careful localization.
-
•
Typography inconsistencies, such as mixing serif and sans-serif fonts, materially impact legibility and conversion rates.
-
•
System fonts improve legibility and page load speed simultaneously, making them a strong choice over fancy custom fonts for UX optimization.
-
•
AB testing platforms that allow granular data slicing, like AB Smartly, empower deeper analysis of user segments and experiment impact.
-
•
Qualitative research is critical to generating valid experiment ideas and interpreting why tests succeed or fail beyond raw data.
-
•
Experiment duration depends heavily on traffic volume, business risk, and the importance of the decision, ranging from one week to several months.
-
•
Guardrail metrics such as customer service calls and repeat bookings help detect negative side effects of experiments beyond conversion.
-
•
Retesting ideas over time is valuable as external conditions, technology, and user contexts evolve, turning previous failures into wins.
Notable Quotes
"I’m actually a really big failure because nine out of 10 tests that we ran failed to show positive impact."
"Most work that most people do there ends up being thrown away."
"Booking.com was averaging more than 2 million room nights booked every day, so the scale is massive."
"Growth is driven by the compound effect where good upon good upon good eventually builds up to fast growth."
"Sometimes people ignore prior test results and test again, and occasionally they find a positive result."
"Execution usually fails, not the concept itself; small things like wrong colors or page load speed sneak in friction."
"Page load time is one of the main killers of conversion."
"I replaced Times New Roman with Arial and it ended up being conclusively and wildly positive for conversions."
"If you don’t need to use fancy typefaces, just don’t, because they can impact page speed and lose conversions."
"Guardrail metrics prevent you from improving conversion at the short term but damaging loyalty or increasing customer service calls."
Or choose a question:
More Videos
"It is imperative that we maintain hope, even when the harshness of reality may suggest the opposite. — Paola Freire"
Angelos ArnisNavigating the Rapid Shifts in Tech's Turbulent Terrain
October 2, 2023
"We are writing the definitive guide because there is no single book dedicated entirely to design ops."
John Calhoun Rachel PosmanBring your DesignOps Story to Life! The Definitive DesignOps Book Jam
October 3, 2023
"If design is not viewed as a player moving the needle on business strategy, designers feel disempowered and frustrated."
Alfred KahnA Seat at the Table: Making Your Team a Strategic Partner
November 29, 2023
"When you accept differences and adjust, you can learn what you’re capable of."
Dan WillisEnterprise Storytelling Sessions
June 3, 2019
"Design is often subordinate to marketing and product management because those functions came first historically."
John Maeda Alison RandAbout Design Organizations (Videoconference)
May 13, 2019
"We had four different supply chains fighting for priority, with different business teams and conflicting systems."
Anat Fintzi Rachel MinnicksDelivering at Scale: Making Traction with Resistant Partners
June 9, 2022
"Nobody wants to buy or use a sloppy product, especially when enterprise users engage daily for hours."
Uday GajendarThe Wicked Craft of Enterprise UX
May 13, 2015
"We’re seeing a shift away from attribution to focus more on impact on the experience itself."
Kristin SkinnerTheme 1 Intro
September 29, 2021
"Service centers were transformed from crowded, threatening environments into spaces that look like five-star hotels with personalized kiosks and family cabanas."
Maish NichaniSparking a Service Excellence Mindset at a Government Agency
December 9, 2021
