Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.
Key Insights
-
•
Conversion design blends design, science, and business to drive intentional, measurable improvements.
-
•
Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.
-
•
Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.
-
•
Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.
-
•
The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.
-
•
Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.
-
•
Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.
-
•
Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.
-
•
Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.
-
•
Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.
Notable Quotes
"Conversion design means to create intentional change."
"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."
"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."
"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."
"You’re never testing an idea purely; you’re always testing the implementation of that idea."
"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."
"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."
"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."
"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"
"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."
Or choose a question:
More Videos
"You can’t just take an entire online world offline, switch it around, and turn it back on; ecosystems must regrow organically."
This Game is Never Done: Design Leadership Techniques from the Video Game World
November 6, 2017
"When times are uncertain, transparency helps tremendously."
Abby CovertStuck? Diagrams Help (Videoconference)
October 27, 2022
"Writing war stories down brings back memories you might have buried deep in your research journey."
Steve Portigal Susan Simon-Daniels Tamara Hale Randolph Duke IIWar Stories LIVE! Q&A-Discussion
March 30, 2020
"There’s no such thing as a self-organized community, in my opinion. They just don’t work."
Kara KaneCommunities of Practice for Civic Design (Videoconference)
April 7, 2022
"Executive buy-in is critical to success but remains a significant challenge for most in the industry."
Caroline VizeThe State of UX: Five Lessons from 2021 to Accelerate Digital Experience in 2022
March 9, 2022
"Sometimes growth is vertical, but we might take different paths and grow sideways and literally move."
Lada GorlenkoTheme 3: Introduction
June 10, 2021
"AI tech debt has compounding interest to it — rushing to market can seriously harm your product and brand."
Jay BustamanteNavigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation
October 2, 2023
"Skipping research is no longer an issue because we empowered designers to build their research skillsets."
Marjorie Stainback Kelsey KingmanTransforming Strategic Research Capacity through Democratization
October 24, 2019
"Design should inherently be ethical and trauma-responsive; trauma-informed should be our default."
Rachael Dietkus, LCSW Uday Gajendar Dr. Dawn Emerick Dawn E. Shedrick, LCSWLeading through the long tail of trauma (Videoconference)
July 13, 2022