Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.
Key Insights
-
•
Conversion design blends design, science, and business to drive intentional, measurable improvements.
-
•
Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.
-
•
Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.
-
•
Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.
-
•
The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.
-
•
Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.
-
•
Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.
-
•
Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.
-
•
Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.
-
•
Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.
Notable Quotes
"Conversion design means to create intentional change."
"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."
"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."
"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."
"You’re never testing an idea purely; you’re always testing the implementation of that idea."
"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."
"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."
"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."
"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"
"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."
Or choose a question:
More Videos
"It is imperative that we maintain hope, even when the harshness of reality may suggest the opposite. — Paola Freire"
Angelos ArnisNavigating the Rapid Shifts in Tech's Turbulent Terrain
October 2, 2023
"You have to grow design ops capabilities in tandem with the design team to avoid organizational mismatches."
John Calhoun Rachel PosmanBring your DesignOps Story to Life! The Definitive DesignOps Book Jam
October 3, 2023
"If design is not viewed as a player moving the needle on business strategy, designers feel disempowered and frustrated."
Alfred KahnA Seat at the Table: Making Your Team a Strategic Partner
November 29, 2023
"No matter how good I think I am at my job, it doesn’t make me a good parent."
Dan WillisEnterprise Storytelling Sessions
June 3, 2019
"You have to give respect to get respect—it's the foundation for connecting people across teams."
John Maeda Alison RandAbout Design Organizations (Videoconference)
May 13, 2019
"Empathy for us needed to be redefined from problem-based to solution-based to avoid an us versus them mindset."
Anat Fintzi Rachel MinnicksDelivering at Scale: Making Traction with Resistant Partners
June 9, 2022
"Craft cultivates customer satisfaction, loyalty, and even forgiveness when functional issues arise."
Uday GajendarThe Wicked Craft of Enterprise UX
May 13, 2015
"This is a super exciting time in the brief history of what we’re calling design operations."
Kristin SkinnerTheme 1 Intro
September 29, 2021
"You might feel like one of the blind men in the story when parts of a process don’t show the whole picture. That’s why you need the full journey exposed."
Maish NichaniSparking a Service Excellence Mindset at a Government Agency
December 9, 2021
