Summary
Mastercard is in the middle of a shift in how decisions are made throughout the product development lifecycle, with data-informed design at the center. A suite of measurements have been developed to quantify the quality of products and the resulting experiences at points across the product lifecycle, so teams can have customer-centric conversations about opportunities and risks. Hear more about these CX Metrics and how we’ve infused them into the product development lifecycle to exert influence across dozens of products across the company.
Key Insights
-
•
MasterCard created CX metrics publicly for the first time to connect design quality with revenue outcomes.
-
•
They use three key scores: Concept Desirability, Design Quality, and Partner Onboarding Gold Standards (POGS).
-
•
Concept Desirability Scores evaluate multiple early ideas cheaply and quickly to identify true opportunities before heavy investment.
-
•
Design Quality Scores focus on a single concept's adoption and detailed user touchpoints including emotional interactions, not just usability.
-
•
POGS measures the quality of implementation touchpoints, aligning incentives around time to revenue and encourages readiness before launch.
-
•
They combat common anti-patterns such as Hippo culture, over-customization, and confusing pilots with real market demand.
-
•
The metrics are integrated into MasterCard’s stage gate product development framework to drive timely course correction and accountability.
-
•
The data from these metrics is designed to be actionable and understandable by product managers, removing the typical research intimidation.
-
•
Human oversight remains crucial to prevent misuse of scores and to ensure metrics supplement traditional research instead of replacing it.
-
•
Leadership buy-in and aligning incentives at the executive level helped embed these metrics into everyday product practices.
Notable Quotes
"We’ve found a way to use data to speak the language of business while tying it back into our values as designers."
"It’s not about launch as success anymore — are customers using it and retaining it? That’s when we make money."
"The Hippo culture is where some important person has an idea without insights, just vibes."
"Willingness to pilot is not the same as willingness to pay — piloting doesn’t mean market demand."
"Concept desirability scores should be run on dozens of ideas fast and cheap early in product development."
"Design quality scores dig deep into all the emotional touchpoints users have with the product, beyond just usability."
"Partner onboarding gold standards are a carrot-shaped stick influencing time to revenue and executive bonuses."
"People feel ownership of the metrics because they help them take action rather than just do research for research’s sake."
"You can’t just run these metrics once — over time, this builds benchmarks and drives continuous improvement."
"Scores aren’t about creating fear — they exist to anchor teams to a bigger vision and encourage real behavior change."
Or choose a question:
More Videos
"If you miss any one of access, accuracy, insight generation, and engagement, it’s not truly democratization."
Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher NashDemocratization: Working with it, not against it [Advancing Research Community Workshop Series]
July 24, 2024
"We ran a design system bootcamp to rotate engineers and prevent bottlenecks and burnout."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"Design principles are guiding tools for conversation rather than strict rules with objective scores."
Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn ResselDiscussion
June 9, 2016
"Adoption is the only thing that matters in innovation — if you’re not changing someone’s behavior, you may have invented something, but you have not innovated."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"Don’t forget the critical change agent persona — adoption and success was determined by the admin or practitioner users."
Malini RaoLessons Learned from a 4-year Product Re-platforming Journey
June 9, 2021
"The System Usability Scale is a powerful benchmarking tool, and reducing it to two questions makes it easier to collect."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"Day one of Design at Scale is going to be about ensuring craft on a massive scale."
Bria AlexanderOpening Remarks
June 9, 2021
"Growth boards demonstrate you can have agile software development at enterprise scale without rigid processes or chaos."
Jackie HoLead Effectively While Preserving Team Autonomy with Growth Boards
January 8, 2024
"Electric car targets for the UK alone would demand more than the world's annual cobalt production."
Dan HillDesigning for the infrastructures of everyday life
June 4, 2024