Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.
Key Insights
-
•
Conversion design blends design, science, and business to drive intentional, measurable improvements.
-
•
Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.
-
•
Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.
-
•
Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.
-
•
The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.
-
•
Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.
-
•
Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.
-
•
Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.
-
•
Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.
-
•
Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.
Notable Quotes
"Conversion design means to create intentional change."
"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."
"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."
"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."
"You’re never testing an idea purely; you’re always testing the implementation of that idea."
"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."
"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."
"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."
"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"
"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."
Or choose a question:
More Videos
"Stakeholder interviews shouldn’t be the only thing democratized—there’s a whole breadth of methodologies to consider."
Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher NashDemocratization: Working with it, not against it [Advancing Research Community Workshop Series]
July 24, 2024
"Design systems are not just UI components but people and culture working together."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"When you have a success, measure time saved and reduction in bugs to build your internal business case for the system."
Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn ResselDiscussion
June 9, 2016
"Key Experience Indicators gave us rigor to bring into engineering conversations to make design less vague or fluffy."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"There isn’t one single formula that works every time for a re-platforming journey."
Malini RaoLessons Learned from a 4-year Product Re-platforming Journey
June 9, 2021
"We use a magic R studio button that, with one click, processes all the survey data automatically."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"You can change the time zone to whatever works for you on the conference site to avoid ambiguity."
Bria AlexanderOpening Remarks
June 9, 2021
"Many product teams have never come face to face with their revenue numbers before."
Jackie HoLead Effectively While Preserving Team Autonomy with Growth Boards
January 8, 2024
"Electric car targets for the UK alone would demand more than the world's annual cobalt production."
Dan HillDesigning for the infrastructures of everyday life
June 4, 2024