Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters

Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO design team represents just 1% of the IT org yet supports UX for 385,000 employees across the whole company.

  • Over half of IBMers have been with the company less than five years, adding urgency to delivering great onboarding and daily experiences.

  • About 30% of IBM employees work remotely or offsite, increasing reliance on intuitive, well-integrated digital tools.

  • The team follows a mission to lead with design, engineer with excellence, deepen agile practices, and secure the enterprise.

  • Kristen’s team moved from reactive user research to building proactive, predictive capabilities by aggregating decades of experience data.

  • Measurement shifted from isolated metrics like NPS to a holistic scorecard including ease of use, capability, and goal completion.

  • A voice of the employee continuous feedback tool replaced large, infrequent surveys to capture real-time user sentiment.

  • Data-driven decisions led to major redesigns of IBM’s employee directory, help system, and corporate intranet, each achieving significant user appreciation.

  • Measurement can be challenging due to resistance and attempts to game survey responses, underscoring the need for data integrity.

  • Embracing design ops chaos and continuous iteration is key to maturing UX capability in a large enterprise organization like IBM.

Notable Quotes

"This circle represents the size of IBM, 385,000 IBMers. The CIO team is only 3% of that, and our design team is just 1% of the CIO."

"We have about 5,000 new IBMers arriving every month, with high expectations and low tolerance for poor experiences."

"The state of IT is a daily reflection of what the company thinks and feels about its employees."

"You can't just jump to the top of the pyramid like Maslow’s hierarchy—you have to consider all the underlying layers supporting the experience."

"Fletcher Prevend took us to the next level by reporting design and user research directly into the CIO office—a huge differentiator."

"Every minute spent struggling with an IT system is a minute not spent delivering value for IBM and our clients."

"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."

"We want to send fewer mailers but make them more impactful, using A/B testing to optimize open and click rates."

"Don’t try to control the data to fit your narrative—the goal is to let insights guide your decisions, not justify them."

"We’re the voice of the employee—we’re arbiters of truth, defenders of experience, and sometimes validators against manipulation."

Ask the Rosenbot
John Calhoun
Have we Reached Our Peak? Spotting the Next Mountain For DesignOps to Climb
2021 • DesignOps Summit 2021
Gold
Dave Gray
Group Activity: Making Sense of DesignOps
2017 • DesignOps Summit 2017
Gold
Phil Gilbert
A Consistent Culture of Design
2015 • Enterprise UX 2015
Gold
Jorge Arango
Meeting of the Waters: Designing for Successful Inorganic Growth
2021 • Enterprise Community
Tess Dixon
C'mon Get Happy
2021 • DesignOps Summit 2021
Gold
Kara Kane
Theme One Intro
2022 • Civic Design 2022
Gold
Louis Rosenfeld
How to use the Rosenbot
2026 • Rosenfeld Community
Barb Spanton
Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
2022 • Design at Scale 2022
Gold
John Maeda
About Design Organizations
2019 • DesignOps Community
Husani Oakley
Theme Three Intro
2023 • Enterprise UX 2023
Gold
Joanna Vodopivec
One Research Team for All - Influence Without Authority
2022 • Advancing Research 2022
Gold
Rachael Dietkus, LCSW
Leading through the long tail of trauma
2022 • Enterprise Community
Christian Crumlish
AMA with Christian Crumlish, author of Product Management for UX People
2022 • Enterprise Community
JJ Kercher
A Roadmap for Maturing Design in the Enterprise
2018 • Enterprise Experience 2018
Gold
Sam Proulx
Understanding Screen Readers on Mobile: How And Why to Learn from Native Users
2023 • Enterprise UX 2023
Gold
Mujtaba Hameed
Frameworks for Excellence: Using Visual Thinking and Communication to Elevate Your Research
2024 • Advancing Research 2024
Gold

More Videos

Xenia Adjoubei

"The Researcher in Residence program invites members of affected communities to use our tools over a funded three-month period pro bono."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Tamara Kartoziia

"When we first started expanding, our internal process felt like being stuck in a massive traffic jam."

Tamara Kartoziia

Think global, adapt local: how service design accelerated B2B market entry by 6 months

November 20, 2025

Jay Bustamante

"You can’t put the toothpaste back in the tube once biased AI harms your brand or users."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Nicole Aleong

"Hope generates optimistic momentum towards a possibility becoming real."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Nick Lewis

"Designing sustainable doesn’t mean removing everything fun from the web."

Nick Lewis

Designing and building low-carbon websites independently

November 18, 2025

Vicky Teinaki

"Having the clients share their screen and having our actual team members watch these live sessions has become very compelling for the entire extended product, UX, and development teams."

Vicky Teinaki Michele Marut Tim Parmee

Short Take #3: UX/Product Lessons from Your Industry Peers

December 6, 2022

Laura Weiss

"Stress makes us stupid because our creative thinking gets cloudy when under pressure."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Joerg Beringer

"Most product teams don't really have user research outputs or they come too late and disconnected from design."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

June 10, 2025

Shelby Switzer

"The nitty gritty is a space to become resources for each other so people know who to talk to when they have questions."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021