Summary
As a product designer, do you trust data when it defies your intuition? What about those times when following data leads you astray? How can you balance the need for experimentation and iteration with the dangers of over-optimization? What are some of the ways that being "data driven" can go wrong? Follow along and learn about the Amplitude design team's journey blending qualitative and quantitative research to drive design strategy—while balancing quality and velocity.
Key Insights
-
•
Designers using product analytics increased by roughly one-third in a year, signalling a shift to data-informed design roles.
-
•
Over-optimizing metrics can backfire, leading teams to improve numbers at the expense of user satisfaction.
-
•
A project ambiguity framework helps determine when and which metrics should guide product initiatives.
-
•
Blue sky (innovative) projects should not be constrained by short-term gating metrics to unleash creativity.
-
•
Aligning qualitative feedback with segmented data strengthens proposals and builds stakeholder confidence.
-
•
Connecting UX improvements via user journeys directly to business outcomes creates a common language for teams.
-
•
Data is not a one-time check but an ongoing feedback loop between research insights and user behavior.
-
•
Misleading data presentations, like altered graph axes, can distort truths and harm decision-making.
-
•
Building relationships with data scientists and sharing joint goals improves collaboration and data use.
-
•
To influence leadership decisions, designers must understand and speak the language of power and metrics relevant to their organization.
Notable Quotes
"It’s a great time to be a designer, and a great time to be making data-driven design decisions."
"We became addicted to our loop of over-optimization, driving metrics that made users frustrated."
"Blue sky projects should have no gating metrics and no short-term metrics."
"We broke down all of our qualitative feedback into different dimensions, which helped lend confidence."
"You can usually draw a convincing dotted line between a UX improvement and a hard business outcome."
"Data is a language you need to learn to speak it."
"Just because it’s a number or chart doesn’t mean it’s true, accurate, or fair."
"Sometimes you lose sight of the forest for the trees when over-optimizing for a single metric."
"Make two teams have the same goal to incentivize collaboration and melt away competing incentives."
"Data is a mindset. It’s never one and done, but a continuous process."
Or choose a question:
More Videos
"Knowledge is power; empowering communities leads to better conservation outcomes."
Alex Hurworth Bonnie John Fahd Arshad Antoine MarinDesigning a Contact Tracing App for Universal Access
October 23, 2020
"We wanted to actively promote the Design Ops discipline to more people, especially those not familiar with our craft."
Laine Riley Prokay Lisa GordonCarving a Path for Early Career DesignOps Practitioners
September 9, 2022
"We stopped talking about patterns and consistency and started talking about scalability and speed to connect with stakeholders."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"Rather than aiming for the chief strategy officer directly, build relationships with junior strategists who are more accessible."
Nathan ShedroffDouble Your Mileage: Use Your Research Strategically
March 31, 2020
"Mobile devices not only have built-in screen readers but also pitch-to-zoom and magnification settings out of the box."
Sam ProulxMobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
November 17, 2022
"Choosing the right cadence depends on your team’s resources and maturity—some do it every two weeks, others monthly."
Feleesha SterlingBuilding a Rapid Research Program (Videoconference)
May 18, 2023
"Bumble’s single feature of women making the first move addressed a huge cultural challenge and lifted the brand."
Neil BarrieWidening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
March 25, 2024
"For many years, folks came to us with questions beyond design — about career paths, tools, skills, and intake management."
John DevanneyThe Design Management Office
November 6, 2017
"Surveys and logs tell us what users do, but qualitative tells us why."
Katy MogalBut Do Your Insights Scale?
March 12, 2021