Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

How to Lead With Data, and Without Data

Gold
Wednesday, June 7, 2023 • Enterprise UX 2023
Share the love for this talk
How to Lead With Data, and Without Data
Speakers: William Newton and Jenny Chang
Link:

Summary

As a product designer, do you trust data when it defies your intuition? What about those times when following data leads you astray? How can you balance the need for experimentation and iteration with the dangers of over-optimization? What are some of the ways that being "data driven" can go wrong? Follow along and learn about the Amplitude design team's journey blending qualitative and quantitative research to drive design strategy—while balancing quality and velocity.

Key Insights

  • Designers are increasingly using data not only to improve UX but to influence product strategy at higher business levels.

  • Too much focus on metrics can lead to over-optimization, harming user experience despite metric improvement.

  • Amplitude developed a framework categorizing projects by ambiguity: blue sky, horizontal, vertical, and paper cuts, guiding the choice of metrics and evaluation.

  • Blue sky projects require no gating short-term metrics because their impact unfolds over longer horizons.

  • Executive buy-in for major initiatives improves when qualitative feedback is segmented and linked to long-term business goals alongside metrics.

  • Redesigning Amplitude’s core chart controls increased user efficiency and conversion by 25%, demonstrating successful data-informed design.

  • Data literacy involves learning the language of metrics relevant to your business and connecting UX improvements to measurable business outcomes.

  • Data must be treated critically: the same data visualized differently can mislead if axes or contexts are adjusted to bias interpretation.

  • Building personal relationships with data scientists and aligning cross-functional goals encourages collaboration and better data use.

  • A continuous feedback loop combining qualitative research and quantitative data strengthens design hypotheses and strategic initiatives.

Notable Quotes

"It’s a great time to be a designer to make data-driven design decisions that impact beyond just pixels."

"We became addicted to our loop of over-optimization, driving traffic to certain features but frustrating users."

"Blue sky projects should have no gating metrics and no short-term metrics because their value is long-term."

"The chart controls were circular and confusing; users looked like deer in headlights not knowing where to start."

"Executives first want to know what metric you will impact next quarter before approving big initiatives."

"Data is a language you need to learn to speak and a mindset that must be continuously practiced."

"You can draw a convincing dotted line between a UX improvement and a hard business outcome if you practice."

"We improved the core chart experience performance by 25%, faster time to save and to first edit have major impact."

"Data is expressive, but just because it’s a chart doesn’t mean the data is true or fair."

"Building personal relationships with data scientists and aligning team goals melts away silos and incentives."

Ask the Rosenbot
Frances Yllana
DesignOps–Leading the Path to Parity
2023 • DesignOps Community
Dolly Parikh
Exit Interview #7: Journey of a Social Entrepreneur
2026 • Rosenfeld Community
Brianna Sylver
Lead With Purpose
2020 • Advancing Research 2020
Gold
Megan Blocker
A Selectively Scrappy Approach to ResearchOps
2018 • DesignOps Summit 2018
Gold
Kaaren Hanson
Stop Talking, Start Doing
2017 • Enterprise Experience 2017
Gold
Michelle Bejian Lotia
Rolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
2023 • Advancing Research 2023
Gold
Amber Knabl
Empowering innovation: The critical role of inclusive product development in the AI era
2024 • Designing with AI 2024
Gold
Joe Meersman
Use AI to Drive Outcomes that Go Beyond the Design Sprint
2024 • DesignOps Summit 2024
Gold
Kelly Goto
Emotion Economy: Ethnography as Corporate Strategy
2015 • Enterprise UX 2015
Gold
Maria Skaaden
Panel Discussion: Methodologies and Work Environments
2018 • DesignOps Summit 2018
Gold
Erin May
Distributed, Democratized, Decentralized: Finding a Research Model to Support Your Org
2022 • Advancing Research 2022
Gold
Sabrina Mach
How to Design Your Design Operating Model
2021 • DesignOps Summit 2021
Gold
Saara Kamppari-Miller
Theme Three Intro
2023 • DesignOps Summit 2023
Gold
Jim Kalbach
Peace is waged with sticky notes: Mapping Real-World Experiences
2018 • Enterprise Experience 2018
Gold
Stefanie Owens
Optimizing for Outcomes: Transformation Design in Systems at Scale
2024 • Advancing Service Design 2024
Gold
Greg Petroff
Design is the Differentiator: Bringing New Design Innovations to a Very Antiquated and Very Large Industry
2021 • Design at Scale 2021
Gold

More Videos

Joanna Vodopivec

"Slack channels with live feeds and tagging let busy developers catch key observations asynchronously."

Joanna Vodopivec Prabhas Pokharel

One Research Team for All - Influence Without Authority

March 9, 2022

Louis Rosenfeld

"You have to build relationships with legal and political leadership to figure out what you can actually change."

Louis Rosenfeld Lashanda Hodge Senongo Akpem Chris Hodowanec

Becoming a Civic Designer: Making the Move from Private to Public Sector

November 17, 2022

Bria Alexander

"We are closing out design ops 2024 with innovation curated by the incomparable John Kuda."

Bria Alexander

Day 3 Welcome

September 25, 2024

Uday Gajendar

"Sponsor sessions are so great that people want recordings. They’re not sales pitches."

Uday Gajendar Louis Rosenfeld

Day 2 Welcome

June 5, 2024

Sam Proulx

"Confidence is a higher burden in retail because people are giving real money; inaccessible flows cause quick abandonment."

Sam Proulx

Online Shopping: Designing an Accessible Experience

June 7, 2023

Russ Unger

"Manhole cover questions belong in the past. Focus on real skills and relevant experience instead."

Russ Unger

Onboarding: The Ecosystem, not the Afterthought

November 7, 2017

Catherine Dubut

"Physical prototyping is a tool to explore interaction modalities and physically connected environments beyond all things digital."

Catherine Dubut

Bridging Physical and Digital Spaces: Approaches to Retail Service Design

March 18, 2021

Josh Clark

"Traditional software is grounded—accurate and reliable—but lacks radical adaptivity."

Josh Clark Veronika Kindred

Sentient Design: New Postures for AI-Mediated Experiences (2nd of 3 seminars)

January 29, 2025

Dantley Davis

"There were things I controlled and things I didn’t, and I needed to plan for both."

Dantley Davis

Leadership & Diversity—A Fireside Chat with Dantley Davis

September 17, 2020