Summary
As a product designer, do you trust data when it defies your intuition? What about those times when following data leads you astray? How can you balance the need for experimentation and iteration with the dangers of over-optimization? What are some of the ways that being "data driven" can go wrong? Follow along and learn about the Amplitude design team's journey blending qualitative and quantitative research to drive design strategy—while balancing quality and velocity.
Key Insights
-
•
Designers using product analytics increased by roughly one-third in a year, signalling a shift to data-informed design roles.
-
•
Over-optimizing metrics can backfire, leading teams to improve numbers at the expense of user satisfaction.
-
•
A project ambiguity framework helps determine when and which metrics should guide product initiatives.
-
•
Blue sky (innovative) projects should not be constrained by short-term gating metrics to unleash creativity.
-
•
Aligning qualitative feedback with segmented data strengthens proposals and builds stakeholder confidence.
-
•
Connecting UX improvements via user journeys directly to business outcomes creates a common language for teams.
-
•
Data is not a one-time check but an ongoing feedback loop between research insights and user behavior.
-
•
Misleading data presentations, like altered graph axes, can distort truths and harm decision-making.
-
•
Building relationships with data scientists and sharing joint goals improves collaboration and data use.
-
•
To influence leadership decisions, designers must understand and speak the language of power and metrics relevant to their organization.
Notable Quotes
"It’s a great time to be a designer, and a great time to be making data-driven design decisions."
"We became addicted to our loop of over-optimization, driving metrics that made users frustrated."
"Blue sky projects should have no gating metrics and no short-term metrics."
"We broke down all of our qualitative feedback into different dimensions, which helped lend confidence."
"You can usually draw a convincing dotted line between a UX improvement and a hard business outcome."
"Data is a language you need to learn to speak it."
"Just because it’s a number or chart doesn’t mean it’s true, accurate, or fair."
"Sometimes you lose sight of the forest for the trees when over-optimizing for a single metric."
"Make two teams have the same goal to incentivize collaboration and melt away competing incentives."
"Data is a mindset. It’s never one and done, but a continuous process."
Dig deeper—ask the Rosenbot:
More Videos
"Amazon performs better on mobile than on desktop according to our Net Promoter Score measurements."
Daniela Magaña Flores Ariane Rahn Jeff Ephraim BanderAhead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
March 10, 2022
"Large language models learn spelling, grammar, how to engage in dialogue, how to write many languages, even code, and even how to argue, persuade, and deceive."
Savina HawkinsHarnessing AI in UXR: Practical Strategies for Positive Impact
March 26, 2024
"When communities are engaged, conservation succeeds dramatically."
Alex Hurworth Bonnie John Fahd Arshad Antoine MarinDesigning a Contact Tracing App for Universal Access
October 23, 2020
"Researchers often struggle to understand why past design decisions were made because knowledge is siloed or buried."
Matt DuignanHITS, Microsoft's internal human insight system: From research library to living body of knowledge (Videoconference)
July 16, 2019
"Make it easy to do the right thing, and as a side effect, make it harder to do the wrong thing - guiding principle from Salesforce Lightning Design System."
Craig VillamorDesign Systems for Ethical Design (Videoconference)
January 26, 2023
"Think of UX as risk mitigation, not a zero-sum game, and share the risk with your stakeholders."
Ovetta SampsonTurning UX Passion into Real Product Influence
June 7, 2023
"If you’re learning something at the end of Sprint six, it’s really costly to go back and make changes in Sprint one or two."
Rima Campbell Amrit S BhachuIncrease Productivity and Drive Business Impact
September 24, 2024
"Find out the root cause why you were hired, because that will make your job easier."
Amy Gawronski ZuccaroAdvice for DesignOps Employee #1
September 29, 2021
"Systems of value determine whose knowledge we increase and whose knowledge we diminish."
Victor UdoewaBeyond Methods and Diversity: The Roots of Inclusion
March 26, 2024
