Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.
Key Insights
-
•
Conversion design blends design, science, and business to drive intentional, measurable improvements.
-
•
Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.
-
•
Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.
-
•
Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.
-
•
The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.
-
•
Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.
-
•
Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.
-
•
Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.
-
•
Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.
-
•
Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.
Notable Quotes
"Conversion design means to create intentional change."
"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."
"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."
"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."
"You’re never testing an idea purely; you’re always testing the implementation of that idea."
"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."
"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."
"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."
"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"
"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."
Or choose a question:
More Videos
"The health of our planet directly influences our health and survival."
Alex Hurworth Bonnie John Fahd Arshad Antoine MarinDesigning a Contact Tracing App for Universal Access
October 23, 2020
"Time commitment for onboarding interns was close to 25%, maybe 50%, of at least two weeks, easing as they became familiar with roles."
Laine Riley Prokay Lisa GordonCarving a Path for Early Career DesignOps Practitioners
September 9, 2022
"If you come with a big idea, they’ll try to dial you back to the smallest iota you can test first."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"Living personas refreshed constantly with new research allow others to connect with real customer behaviors."
Nathan ShedroffDouble Your Mileage: Use Your Research Strategically
March 31, 2020
"Mobile browsers are typically updated with the OS, reducing variability compared to desktop browser versions."
Sam ProulxMobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
November 17, 2022
"Foundational research answers big, nebulous questions; rapid research focuses on specific usability questions."
Feleesha SterlingBuilding a Rapid Research Program (Videoconference)
May 18, 2023
"Overlaying cultural dialogue on journey maps reveals impactful insights that need to be validated but are promising."
Neil BarrieWidening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
March 25, 2024
"Knowledge management isn’t just about preserving work, it’s about retaining motivated, growing people."
John DevanneyThe Design Management Office
November 6, 2017
"Find the curious partners who will do some of the evangelizing for you to their peers."
Katy MogalBut Do Your Insights Scale?
March 12, 2021