Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.
Key Insights
-
•
Conversion design blends design, science, and business to drive intentional, measurable improvements.
-
•
Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.
-
•
Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.
-
•
Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.
-
•
The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.
-
•
Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.
-
•
Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.
-
•
Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.
-
•
Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.
-
•
Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.
Notable Quotes
"Conversion design means to create intentional change."
"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."
"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."
"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."
"You’re never testing an idea purely; you’re always testing the implementation of that idea."
"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."
"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."
"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."
"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"
"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."
Or choose a question:
More Videos
"Frequency and memorability in staff training is critical for accessible in-person service."
Sam ProulxOnline Shopping: Designing an Accessible Experience
June 7, 2023
"Participatory design is not what you think it is—it’s about taking information, generating ideas, and testing them out collaboratively."
Bria Alexander Ariel Kennan Charlotte Lee Sarah Brooks Emily Lessard Gordon Ross Joanne DongReflect and Chart Forward
December 10, 2021
"Service designers connect the dots, the elements within systems that need to align for the business to run."
Gina MendoliaTherapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
December 3, 2024
"The interface becomes a radically adaptive surface, an intelligent canvas that responds to your needs in context."
Josh Clark Veronika KindredSentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
January 15, 2025
"Fear of the unknown during change causes physical, emotional, and psychological stress."
Alla WeinbergPeople Are Sick of Change: Psychological Safety is the Cure (Videoconference)
July 20, 2023
"Research is like a compass—it provides direction, not a detailed map with all the turns and dangers ahead."
Cassini Nazir Meah LinThe Dangers of Empathy: Toward More Responsible Design Research
March 27, 2023
"Surprisingly, onboarding for designers was basically ‘here’s your laptop, good luck.’ We needed to fix that."
Brennan HartichCommunicating and Establishing DesignOps as a New Function
November 7, 2018
"These changes were effective immediately — going from 80 to 170 people overnight was a shock but also an opportunity."
Isaac HeyveldExpand DesignOps Leadership as a Chief of Staff
September 8, 2022
"A culture of care is built on small acts of trust, like checking in about someone's weekend or helping with personal challenges."
Etienne FangThe Power of Care: From Human-Centered Research to Humanity-Centered Leadership
March 10, 2021
