Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group (Rosenfeld Community)
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • Standard metrics like NPS often mask important differences in user experience across customer segments such as new and existing customers.

  • Metrics should separate attitudinal data (how users feel) from behavioral data (what users do) to avoid conflation and misinterpretations.

  • A custom digital metric framework benefits from integrating qualitative sentiment analysis of user verbatims to add context beyond numbers.

  • Balancing metric complexity internally with simplified reporting for business stakeholders helps drive actionable insights without overwhelming decision makers.

  • Applying established frameworks like Google HEART or NASA TLX can provide helpful reference points for measuring user experience multidimensionally.

  • Metrics need to be valid over time, allowing benchmarking across consistent, relative timeframes to detect meaningful trends.

  • Measuring customer sentiment includes factors like perceived trustworthiness, visual appeal, clarity, and satisfaction along specific user journeys.

  • Conversion rates alone don't tell the full story—total conversions and behavioral context must be considered when evaluating performance.

  • Business leaders often prefer single-number metrics, so combining simple scores with human stories and verbatim feedback aids adoption.

  • Designing metrics starts from defining clear goals, followed by specifying measure type, signals, timeframe, and scope for each metric.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex and behavior is even more complex."

"The average NPS score might rise while some key customer groups actually have declining satisfaction."

"A metric is usually the result of a relation between two measures, not to be confused with the raw measurement itself."

"People tend to inflate negative feelings in surveys which may not reflect their actual behavior."

"We wanted the metric to be simple for users to complete, so we chose a five-point Likert scale."

"It's easy for business to focus on moving a single needle rather than juggling multiple complex factors."

"Stories and verbatims help drive the point home alongside metrics to make the results more meaningful for stakeholders."

"Google HEART is a great framework but difficult to implement fully because it relies on many signals that aren't always available."

"Metrics should be relative to a time period and benchmarked over time to retain their meaning and validity."

"Designing a metric starts with clear definitions of quantity, signals, timeframe, and scope."

Ask the Rosenbot
Andreas Huebner
What Is It Like To Be Part of The UX Team at Compass?
2021 • Advancing Research 2021
Gold
Kristen Honey
"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design (Videoconference)
2021 • Civic Design Community
Satyam Kantamneni
Do You Have an Experience Vision? (Videoconference)
2023 • Enterprise Community
Farid Sabitov
Theme Four Intro
2022 • DesignOps Summit 2022
Gold
Kayla Farrell
What It's Like To Be a User Researcher at Compass
2021 • Advancing Research 2021
Gold
Louis Rosenfeld
Discussion: What Operations can teach DesignOps
2017 • DesignOps Summit 2017
Gold
Sarah Brooks
Theme 3 Intro
2021 • Civic Design 2021
Gold
Noel Lamb
Cultivating Business Partnerships to Grow Research Ops (Videoconference)
2022 • Advancing Research Community
Anna Avrekh
Diversity In and For Design: Building Conscious Diversity in Design and Research
2021 • Design at Scale 2021
Gold
Sarah Fathallah
A Typology of Participation in Participatory Research
2023 • Advancing Research 2023
Gold
Steve Portigal
Looking Back…to Look Ahead
2024 • Advancing Research 2024
Gold
Carol Massa
Designing Health: Integrating Service Design, Technology, and Strategy to Transform Patient and Clinician Experiences
2024 • Advancing Service Design 2024
Gold
Laureen Kattan
Centering Patients and Clinicians in a Complex Government Ecosystem
2023 • Design in Product 2023
Gold
George Abraham
Design Systems To-Go: Introducing a Starter Design System, and Indigo.Design Overview (Part 1)
2021 • DesignOps Summit 2021
Gold
Magdalena Zadara
Zero Hour: How to Get Far Quickly When Starting Your Digital Service Unit Late
2022 • Civic Design 2022
Gold
Shipra Kayan
How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
2021 • DesignOps Summit 2021
Gold

More Videos

Jennifer Kanyamibwa

"Telling your team’s story explains the culture of your team and how someone could fit into it."

Jennifer Kanyamibwa

Creating the Blueprint: Growing and Building Design Teams

November 8, 2018

Brigette Metzler

"If you put more in than you take out of a repository, that’s a good rule of thumb."

Brigette Metzler Dana Chrisfield

Research Repositories: A global project by the ResearchOps Community (Videoconference)

August 27, 2020

Carl Turner

"We converted process boxes directly into user stories to create a common language for the team."

Carl Turner

You Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project

March 28, 2023

John Mortimer

"Often we designers are in IT departments, separate from the larger organization; that boundary is a hurdle to overcome."

John Mortimer Milan Guenther Lucy Ellis Patrick Quattlebaum

Panel Discussion

December 3, 2024

Dante Guintu

"Gallup says only twelve percent strongly agree their onboarding is great; that’s an abysmal metric."

Dante Guintu

How to Crush the Talent Crunch

September 8, 2022

Richard Buchanan

"The new marketing is about preparing users or customers for the new kinds of products that will emerge as time moves forward."

Richard Buchanan

Creativity and Principles in the Flourishing Enterprise

June 15, 2018

Dan Willis

"Many of us have worked as outsiders, insiders, and then outsiders again throughout our careers."

Dan Willis

Theme 3: Intro

January 8, 2024

Dan Ward

"I may have failed, but I am not a failure."

Dan Ward

Failure Friday #1 with Dan Ward

February 7, 2025

Chris Geison

"We need to step out from informing decisions and into becoming changemakers."

Chris Geison

Theme Two Intro

March 28, 2023