Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters
Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO IT department is only 3% of the company but manages IT for 12,000 IBMers; the CIO design team is just 1% of that group.

  • Over half of IBM’s 385,000 employees have been hired within five years, introducing ongoing challenges in onboarding and experience design.

  • 30% of IBMers work remotely or offsite, necessitating resilient, intuitive digital tools without onsite support.

  • The team uses an IT pyramid of pain inspired by Maslow’s hierarchy to ensure foundational tech and network reliability before focusing on tasks.

  • Continuous measurement includes tracking NPS, goal completion, ease of use, and user comments linked directly to employee IDs for rich insights.

  • IBM shifted from infrequent large surveys to daily embedded Voice of the Employee forms capturing immediate feedback on 4,000 submissions/month.

  • The design ops team integrates research, UX, content, and coaching to foster collaboration and prevent silos.

  • A predictive AI model helps identify when employees should upgrade their hardware before critical failures occur, leading IBM to offer on-demand new machines.

  • Transparency in metrics drives improvement and accountability, with NPS trending to be incorporated into employee evaluations starting next year.

  • Kristen emphasizes embracing chaos and iteration in design ops rather than forcing rigid processes, highlighting the team's evolution from phase one simplicity to complex maturity.

Notable Quotes

"This small but mighty CIO design team is literally 1% of the 3% IT population."

"Over half of IBMers have been here less than five years, adding complexity to designing their experience."

"The state of IT is a daily reflection of what IBM thinks and feels about its employees."

"You can’t just focus on the task level, you have to consider the entire ecosystem and its layers."

"People take out a pen when the CIO says I have a design user experience and research function reporting directly to me."

"We’re here to make work better, and the coaching tribe is here to make teams better."

"We moved from large infrequent surveys to daily Voice of the Employee forms capturing real-time user sentiment."

"Every minute spent struggling with an IT system is a minute not delivering value to IBM and our clients."

"Don’t try to control or force data to say what you want. The goal is for data to help make the right decision."

"Measurement matters. Even from a small circle, you can make a big impact."

Ask the Rosenbot
Saara Kamppari-Miller
Theme Three Intro
2023 • DesignOps Summit 2023
Gold
Ross Smith
Breaking Barriers with Empathy
2017 • Enterprise Experience 2017
Gold
Jilanna Wilson
Distributed Design Operations Management
2019 • DesignOps Summit 2019
Gold
Brendan Jarvis
It was the Best of Times. It was the Worst of Times.
2024 • DesignOps 2024
Gold
Sabrina Mach
How to Design Your Design Operating Model
2021 • DesignOps Summit 2021
Gold
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold
Verónica Urzúa
The B-side of the Research Impact
2021 • Advancing Research 2021
Gold
Abby Covert
Panel: Collaboration Tools
2017 • DesignOps Summit 2017
Gold
Emilia Åström
Unlock Your Team’s Intelligence with Collaboration Design
2022 • Design at Scale 2022
Gold
Emily Eagle
Can't Rewind: Radio and Retail
2019 • Enterprise Experience 2019
Gold
Sahibzada Mayed
The Politics of Radical Research: A Manifesto
2023 • Advancing Research 2023
Gold
Ari Zelmanow
Dark Side of Democratization (Videoconference)
2023 • Advancing Research Community
Josina Vink
Navigating the pitfalls of systems thinking in service design
2024 • Advancing Service Design 2024
Gold
Sofía Delsordo
Public Policy for Jalisco's Designers to Make Design Matter
2021 • Civic Design 2021
Gold
Bria Alexander
Welcome
2024 • Enterprise Experience 2020
Gold
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold

More Videos

Alex Hurworth

"Innovation in conservation practices is desperately needed to keep pace with environmental changes."

Alex Hurworth Bonnie John Fahd Arshad Antoine Marin

Designing a Contact Tracing App for Universal Access

October 23, 2020

Laine Riley Prokay

"Preparation is critical to ensure new employees have everything they need prior to their first day and the first few weeks."

Laine Riley Prokay Lisa Gordon

Carving a Path for Early Career DesignOps Practitioners

September 9, 2022

Eniola Oluwole

"Everyone wanted to know what was the official pattern and who was accountable for it."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Nathan Shedroff

"Design researchers know customers better than almost anyone else in the organization, yet they are rarely invited into strategy processes."

Nathan Shedroff

Double Your Mileage: Use Your Research Strategically

March 31, 2020

Sam Proulx

"Accessibility is an ongoing process where we iterate, improve, and expand, and mobile-first makes the journey easier."

Sam Proulx

Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World

November 17, 2022

Feleesha Sterling

"Documentation and making findings visible even without a dedicated research librarian helps maintain program momentum."

Feleesha Sterling

Building a Rapid Research Program (Videoconference)

May 18, 2023

Neil Barrie

"Overlaying cultural dialogue on journey maps reveals impactful insights that need to be validated but are promising."

Neil Barrie

Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture

March 25, 2024

John Devanney

"Incremental improvements and disruptive innovation require very different methods and measures."

John Devanney

The Design Management Office

November 6, 2017

Katy Mogal

"Surveys and logs tell us what users do, but qualitative tells us why."

Katy Mogal

But Do Your Insights Scale?

March 12, 2021