Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Get Your Whole Team Testing to Design for Impact
Wednesday, July 24, 2024 • Rosenfeld Community
Share the love for this talk
Get Your Whole Team Testing to Design for Impact
Speakers: Erin Weigel
Link:

Summary

Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.

Key Insights

  • Conversion design blends design, science, and business to drive intentional, measurable improvements.

  • Early 20th-century agricultural experiments lacked proper controls, inspiring Ronald Fisher's foundational Design of Experiments.

  • Conversion means change, not just sales or monetary gain, emphasizing meaningful improvement over mere difference.

  • Traditional linear product development lacks the dynamic feedback and iteration inherent in systems thinking.

  • The conversion design process is seven interconnected phases that often loop back rather than proceed linearly.

  • Good experimentation requires upfront hypothesis documentation to reduce bias and ensure reliable data.

  • Randomized 50/50 A/B testing remains the gold standard for isolating cause and effect in product experiments.

  • Prioritizing experiments across multiple buckets—including bug fixes and pricing strategies—avoids over-focusing on just content tweaks.

  • Ethical considerations, including non-discrimination and long-term sustainability, are essential in design decision-making.

  • Building an experimentation culture compounds business value and collective knowledge, improving decision quality over time.

Notable Quotes

"Conversion design means to create intentional change."

"Design is the rendering of intent — bringing ideas into a form people can interact with to solve problems, as Jared Spool said."

"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."

"Decades worth of agricultural data had to be thrown out because they lacked control groups and statistical rigor."

"You’re never testing an idea purely; you’re always testing the implementation of that idea."

"Randomization is magic — it evenly distributes confounds so the only difference affecting results is your change."

"A lot of developers are way too confident they write perfect code; testing bug fixes often reveals hidden issues."

"Ethics evolve faster than law; just because something is legal doesn’t mean it’s ethical."

"If my experiment made front-page news tomorrow, what would the headline be? Would my mother be proud?"

"The conversion design process creates collective knowledge, which gets reinfused to strengthen future experiments."

Ask the Rosenbot
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Matt Duignan
HITS, Microsoft's internal human insight system: From research library to living body of knowledge (Videoconference)
2019 • Advancing Research Community
Mandy Drew
What Role(s) Can Research Play in Responsible Design?
2021 • Advancing Research 2021
Gold
Product and Design at Bloomberg: A 15-year Evolution
2022 • Design in Product 2022
Gold
Victor Lombardi
Bridging Design and Climate Science (Videoconference)
2024 • Climate UX Interest Group (Rosenfeld Community)
Melinda Belcher
Bridging the Gap: Making the Most of the Differences Between Agency and Enterprise
2024 • Enterprise Experience 2020
Gold
Brad Orego
Bringing Customer Research to More Internal Teams
2022 • Advancing Research 2022
Gold
Louis Rosenfeld
Discussion: What Operations can teach DesignOps
2017 • DesignOps Summit 2017
Gold
Dave Malouf
The Future of DesignOps
2024 • DesignOps Summit 2020
Gold
Dane DeSutter
Keeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
2024 • Advancing Research 2024
Gold
Kristin Taylor
Building Bridges Across Organizational Silos
2022 • Civic Design 2022
Gold
Sam Proulx
Understanding Screen Readers on Mobile: How And Why to Learn from Native Users
2023 • Enterprise UX 2023
Gold
Boon Yew Chew
Making Sense of Systems—and Using Systems to Make Sense of the Enterprise
2023 • Enterprise UX 2023
Gold
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Tess Dixon
C'mon Get Happy
2021 • DesignOps Summit 2021
Gold
Erin Hauber
Design is Not the Frosting on the Scaled Agile Layer Cake
2019 • DesignOps Summit 2019
Gold

More Videos

Alex Hurworth

"The health of our planet directly influences our health and survival."

Alex Hurworth Bonnie John Fahd Arshad Antoine Marin

Designing a Contact Tracing App for Universal Access

October 23, 2020

Laine Riley Prokay

"Time commitment for onboarding interns was close to 25%, maybe 50%, of at least two weeks, easing as they became familiar with roles."

Laine Riley Prokay Lisa Gordon

Carving a Path for Early Career DesignOps Practitioners

September 9, 2022

Eniola Oluwole

"If you come with a big idea, they’ll try to dial you back to the smallest iota you can test first."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Nathan Shedroff

"Living personas refreshed constantly with new research allow others to connect with real customer behaviors."

Nathan Shedroff

Double Your Mileage: Use Your Research Strategically

March 31, 2020

Sam Proulx

"Mobile browsers are typically updated with the OS, reducing variability compared to desktop browser versions."

Sam Proulx

Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World

November 17, 2022

Feleesha Sterling

"Foundational research answers big, nebulous questions; rapid research focuses on specific usability questions."

Feleesha Sterling

Building a Rapid Research Program (Videoconference)

May 18, 2023

Neil Barrie

"Overlaying cultural dialogue on journey maps reveals impactful insights that need to be validated but are promising."

Neil Barrie

Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture

March 25, 2024

John Devanney

"Knowledge management isn’t just about preserving work, it’s about retaining motivated, growing people."

John Devanney

The Design Management Office

November 6, 2017

Katy Mogal

"Find the curious partners who will do some of the evangelizing for you to their peers."

Katy Mogal

But Do Your Insights Scale?

March 12, 2021