Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
UX Lessons from running more than 1,200 A/B Tests
Summary
Key Insights
-
•
Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.
-
•
Execution often fails on good ideas rather than the ideas themselves.
-
•
Technical details like page load time can negate positive design changes if not carefully managed.
-
•
Edge case bugs in different languages or currencies can silently kill conversions.
-
•
Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.
-
•
Using system fonts improves legibility and page load speed, positively affecting conversions.
-
•
Tracking should only include users exposed to the tested change to avoid noisy data.
-
•
Qualitative user research drives many experiment ideas by revealing real user challenges.
-
•
Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.
-
•
Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.
Notable Quotes
"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."
"Compound effect means good upon good upon good eventually builds to incredibly fast growth."
"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."
"Increasing page load time by three seconds can nullify all your design improvements."
"Picking the wrong size image can impact customer experience as much as what you see on screen."
"There are as many versions of the website as stars in the sky because of language, currency, and device variations."
"No tracking is perfect because numbers tell you what likely happened, but not why it happened."
"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."
"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."
"You can’t fail 90 times out of 10 without laughing at yourself to keep going."
Or choose a question:
More Videos
"Going from 100 to 200 people, suddenly product managers started hearing from a dozen people a week with conflicting requests."
Shipra KayanHow we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
September 30, 2021
"When we understand organizations as ecosystems, we recognize the need for organic simplicity."
Peter MorvilleThe Architecture of Understanding
May 13, 2015
"If you make a pretty page, that’s great. But if the users can’t actually use the page or they’re frustrated, you just have a pretty page."
Renee ReidBecoming a ResearchH.E.R (Highly Enterprise Ready)
June 3, 2019
"You don’t need special software to get started analyzing gestures and embodied actions; you just need video and a notepad."
Dane DeSutterKeeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
March 26, 2024
"Sometimes accessibility isn’t perfect, but consistency lets me shop on Amazon in my sleep because of muscle memory."
Sam ProulxOnline Shopping: Designing an Accessible Experience
October 3, 2023
"Sometimes vocal feedback reveals problems that digital behavior can’t see—like offline factors affecting perception."
Andrew Custage Michael MallettThe Digital Journey: Research on Consumer Frustration and Loyalty
March 29, 2023
"The primary goal of a hackathon is the community building aspect and getting people to talk to each other and learn from each other."
Shelby SwitzerMaking Space for Community Knowledge-sharing in a Distributed World
December 10, 2021
"Volume is a losing game because we cannot outcompete full AI automation for sheer speed and quantity of insights."
Llewyn PaineCoexisting with AI: A practical guide for researchers to navigate tools, ethics, and integration
March 11, 2025
"Researchers make customers visible inside the organization, but also help the organization learn about itself."
Robin BeersNavigating organizational systems: Rethinking researcher’s role in driving change
March 13, 2025
