Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Get Your Whole Team Testing to Design for Impact
Wednesday, July 24, 2024 • Rosenfeld Community
Share the love for this talk
Get Your Whole Team Testing to Design for Impact
Speakers: Erin Weigel
Link:

Summary

Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.

Key Insights

  • Conversion design blends design, science, and business to drive intentional, measurable improvements.

  • Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.

  • Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.

  • Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.

  • The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.

  • Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.

  • Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.

  • Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.

  • Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.

  • Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.

Notable Quotes

"Conversion design means to create intentional change."

"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."

"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."

"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."

"You’re never testing an idea purely; you’re always testing the implementation of that idea."

"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."

"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."

"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."

"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"

"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."

Ask the Rosenbot
Greg Petroff
Software as Material—A Redux
2023 • Enterprise UX 2023
Gold
Sean Dolan
A Practical Look at Creating More Usable Enterprise Customer Journeys (Videoconference)
2019 • Enterprise Community
Louis Rosenfeld
Do you want to work on climate? (Climate UX Discussion Series)
2023 • Climate UX Interest Group (Rosenfeld Community)
Leah Buley
Closing Plenary: The Crisis of Digital
2020 • Advancing Research 2020
Gold
Sarah Barrett
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
2025 • Rosenfeld Community
Alana Washington
Theme 3 Intro
2021 • DesignOps Summit 2021
Gold
Lona Moore
Scaling Design Beyond Designers
2021 • Design at Scale 2021
Gold
Sarah Williams
Verizon_A Framework for CX Transformation
2024 • Design at Scale 2021
Gold
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Design at Scale 2022
Gold
Pippa Lomas
Paving the Path for Neurodiversity in Design
2023 • DesignOps Summit 2023
Gold
Dave Hora
Advice for Establishing Research (Videoconference)
2022 • Advancing Research Community
Ari Zelmanow
Dark Side of Democratization (Videoconference)
2023 • Advancing Research Community
Alicia Mooty
Design Staffing Models
2021 • DesignOps Summit 2021
Gold
Stephen Anderson
Puzzled? How to Coordinate Humans for Complex Challenges
2021 • Enterprise Community
Steve Portigal
War Stories LIVE! Steve Portigal
2020 • Advancing Research 2020
Gold
Sarah Alvarado
How to make UX research leadership more effective [Advancing Research Community Workshop Series] (Videoconference)
2023 • Advancing Research Community

More Videos

Jemma Ahmed

"Stakeholder interviews shouldn’t be the only thing democratized—there’s a whole breadth of methodologies to consider."

Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher Nash

Democratization: Working with it, not against it [Advancing Research Community Workshop Series]

July 24, 2024

Nina Jurcic

"Design systems are not just UI components but people and culture working together."

Nina Jurcic

The Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change

October 3, 2023

Nathan Curtis

"When you have a success, measure time saved and reduction in bugs to build your internal business case for the system."

Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn Ressel

Discussion

June 9, 2016

Saara Kamppari-Miller

"Key Experience Indicators gave us rigor to bring into engineering conversations to make design less vague or fluffy."

Saara Kamppari-Miller Nicole Bergstrom Shashi Jain

Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”

November 13, 2024

Malini Rao

"There isn’t one single formula that works every time for a re-platforming journey."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021

Mackenzie Cockram

"We use a magic R studio button that, with one click, processes all the survey data automatically."

Mackenzie Cockram Sara Branco Cunha Ian Franklin

Integrating Qualitative and Quantitative Research from Discovery to Live

December 16, 2022

Bria Alexander

"You can change the time zone to whatever works for you on the conference site to avoid ambiguity."

Bria Alexander

Opening Remarks

June 9, 2021

Jackie Ho

"Many product teams have never come face to face with their revenue numbers before."

Jackie Ho

Lead Effectively While Preserving Team Autonomy with Growth Boards

January 8, 2024

Dan Hill

"Electric car targets for the UK alone would demand more than the world's annual cobalt production."

Dan Hill

Designing for the infrastructures of everyday life

June 4, 2024