Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Driving Change with CX Metrics
Gold
Wednesday, June 7, 2023 • Enterprise UX 2023
Share the love for this talk
Driving Change with CX Metrics
Speakers: Todd Healy and Jess Greco
Link:

Summary

Mastercard is in the middle of a shift in how decisions are made throughout the product development lifecycle, with data-informed design at the center. A suite of measurements have been developed to quantify the quality of products and the resulting experiences at points across the product lifecycle, so teams can have customer-centric conversations about opportunities and risks. Hear more about these CX Metrics and how we’ve infused them into the product development lifecycle to exert influence across dozens of products across the company.

Key Insights

  • MasterCard created CX metrics publicly for the first time to connect design quality with revenue outcomes.

  • They use three key scores: Concept Desirability, Design Quality, and Partner Onboarding Gold Standards (POGS).

  • Concept Desirability Scores evaluate multiple early ideas cheaply and quickly to identify true opportunities before heavy investment.

  • Design Quality Scores focus on a single concept's adoption and detailed user touchpoints including emotional interactions, not just usability.

  • POGS measures the quality of implementation touchpoints, aligning incentives around time to revenue and encourages readiness before launch.

  • They combat common anti-patterns such as Hippo culture, over-customization, and confusing pilots with real market demand.

  • The metrics are integrated into MasterCard’s stage gate product development framework to drive timely course correction and accountability.

  • The data from these metrics is designed to be actionable and understandable by product managers, removing the typical research intimidation.

  • Human oversight remains crucial to prevent misuse of scores and to ensure metrics supplement traditional research instead of replacing it.

  • Leadership buy-in and aligning incentives at the executive level helped embed these metrics into everyday product practices.

Notable Quotes

"We’ve found a way to use data to speak the language of business while tying it back into our values as designers."

"It’s not about launch as success anymore — are customers using it and retaining it? That’s when we make money."

"The Hippo culture is where some important person has an idea without insights, just vibes."

"Willingness to pilot is not the same as willingness to pay — piloting doesn’t mean market demand."

"Concept desirability scores should be run on dozens of ideas fast and cheap early in product development."

"Design quality scores dig deep into all the emotional touchpoints users have with the product, beyond just usability."

"Partner onboarding gold standards are a carrot-shaped stick influencing time to revenue and executive bonuses."

"People feel ownership of the metrics because they help them take action rather than just do research for research’s sake."

"You can’t just run these metrics once — over time, this builds benchmarks and drives continuous improvement."

"Scores aren’t about creating fear — they exist to anchor teams to a bigger vision and encourage real behavior change."

Ask the Rosenbot
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Advancing Research 2022
Gold
Shanti Mathew
Civic Design at Scale: Introducing the Public Policy Layer Cake
2021 • Civic Design 2021
Gold
Bria Alexander
Opening Remarks
2022 • Civic Design 2022
Gold
Dr. Jamika D. Burge
Broad Strokes: Connecting Design, Research, and AI to the World Around Us
2023 • Enterprise UX 2023
Gold
Etienne Fang
Power of Insights: Why sharing is better than silos with Uber’s Insights Platform (Videoconference)
2019 • Advancing Research Community
John Cutler
Prioritization for designers and product managers (1st of 3 seminars) (Videoconference)
2024 • Rosenfeld Community
Shipra Kayan
How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
2021 • DesignOps Summit 2021
Gold
Mitchell Bernstein
Organizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape
2021 • DesignOps Summit 2021
Gold
Bria Alexander
OKRs—Helpful or Harmful? (Videoconference)
2022 • DesignOps Community
Mila Kuznetsova
How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices
2022 • Advancing Research 2022
Gold
Kit Unger
Theme 2: Introduction
2021 • Design at Scale 2021
Gold
Jon Fukuda
Storytelling for DesignOps (Videoconference)
2023 • DesignOps Community
Mike Oren
Design Research Strategy & Strategic Design Research (Videoconference)
2022 • Advancing Research Community
Dave Malouf
Closing Keynote: Amplify. Not Optimize.
2019 • DesignOps Summit 2019
Gold
Theresa Slate
Why Changing Hearts & Minds Doesn’t Work When Promoting DE&I Efforts, but Checklists Do
2023 • DesignOps Summit 2023
Gold
Kim Holt
A Salesforce Panel Discussion on Values-Driven DesignOps
2022 • DesignOps Summit 2022
Gold

More Videos

Jennifer Kanyamibwa

"We started doing something interesting: every Monday, instead of having a stand-up, we would journal and share."

Jennifer Kanyamibwa

Creating the Blueprint: Growing and Building Design Teams

November 8, 2018

Brigette Metzler

"Most organizations are using a stack of tools like Google Drive, SharePoint, or Airtable, not mature repositories."

Brigette Metzler Dana Chrisfield

Research Repositories: A global project by the ResearchOps Community (Videoconference)

August 27, 2020

Carl Turner

"Hardest to uncover are the unconscious assumptions people make about how things should be done."

Carl Turner

You Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project

March 28, 2023

John Mortimer

"Trust, facilitation, change, and complexity—if you wrap your mind around those, you’ll get reasonably far."

John Mortimer Milan Guenther Lucy Ellis Patrick Quattlebaum

Panel Discussion

December 3, 2024

Dante Guintu

"Gallup says only twelve percent strongly agree their onboarding is great; that’s an abysmal metric."

Dante Guintu

How to Crush the Talent Crunch

September 8, 2022

Richard Buchanan

"Intuition is the ability to see a connection so important that you could build a system out of it."

Richard Buchanan

Creativity and Principles in the Flourishing Enterprise

June 15, 2018

Dan Willis

"Every speaker before me had these pristine backgrounds. I’m a slob and feeling really self-conscious about it."

Dan Willis

Theme 3: Intro

January 8, 2024

Dan Ward

"Failure is when effort did not produce the desired result, not an identity."

Dan Ward

Failure Friday #1 with Dan Ward

February 7, 2025

Chris Geison

"The talks today start more societal and foundational and end with advancing our day-to-day research practice."

Chris Geison

Theme Two Intro

March 28, 2023