Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters
Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO IT department is only 3% of the company but manages IT for 12,000 IBMers; the CIO design team is just 1% of that group.

  • Over half of IBM’s 385,000 employees have been hired within five years, introducing ongoing challenges in onboarding and experience design.

  • 30% of IBMers work remotely or offsite, necessitating resilient, intuitive digital tools without onsite support.

  • The team uses an IT pyramid of pain inspired by Maslow’s hierarchy to ensure foundational tech and network reliability before focusing on tasks.

  • Continuous measurement includes tracking NPS, goal completion, ease of use, and user comments linked directly to employee IDs for rich insights.

  • IBM shifted from infrequent large surveys to daily embedded Voice of the Employee forms capturing immediate feedback on 4,000 submissions/month.

  • The design ops team integrates research, UX, content, and coaching to foster collaboration and prevent silos.

  • A predictive AI model helps identify when employees should upgrade their hardware before critical failures occur, leading IBM to offer on-demand new machines.

  • Transparency in metrics drives improvement and accountability, with NPS trending to be incorporated into employee evaluations starting next year.

  • Kristen emphasizes embracing chaos and iteration in design ops rather than forcing rigid processes, highlighting the team's evolution from phase one simplicity to complex maturity.

Notable Quotes

"This small but mighty CIO design team is literally 1% of the 3% IT population."

"Over half of IBMers have been here less than five years, adding complexity to designing their experience."

"The state of IT is a daily reflection of what IBM thinks and feels about its employees."

"You can’t just focus on the task level, you have to consider the entire ecosystem and its layers."

"People take out a pen when the CIO says I have a design user experience and research function reporting directly to me."

"We’re here to make work better, and the coaching tribe is here to make teams better."

"We moved from large infrequent surveys to daily Voice of the Employee forms capturing real-time user sentiment."

"Every minute spent struggling with an IT system is a minute not delivering value to IBM and our clients."

"Don’t try to control or force data to say what you want. The goal is for data to help make the right decision."

"Measurement matters. Even from a small circle, you can make a big impact."

Ask the Rosenbot
Alicia D. Johnson
Disasters and the 21st Century
2021 • Civic Design 2021
Gold
Susan Weinschenk
Evaluating the Maturity of UX in Your Organization (Videoconference)
2020 • Enterprise Community
Alissa Briggs
How to Coach Enterprise Experimentation
2015 • Enterprise UX 2015
Gold
Rittika Basu
Age and Interfaces: Equipping Older Adults with Technological Tools (Videoconference)
2023 • Advancing Research Community
Maria Skaaden
Continuous Design: One eye on the horizon and the other on the next wave
2018 • DesignOps Summit 2018
Gold
Onur Kocan
Understanding the Strategy for Civic Design in a Complex City: Istanbul
2022 • Civic Design 2022
Gold
Renee Reid
Becoming a ResearchH.E.R (Highly Enterprise Ready)
2019 • Enterprise Experience 2019
Gold
John Calhoun
Bring your DesignOps Story to Life! The Definitive DesignOps Book Jam
2023 • DesignOps Summit 2023
Gold
Ned Dwyer
The Intersection of Design and ResearchOps
2024 • DesignOps 2024
Gold
Asia Hoe
Partnering with Product: A Journey from Junior to Senior Design
2023 • Design in Product 2023
Gold
Billy Carlson
Pro-level UI Tips for Beginners
2022 • DesignOps Summit 2022
Gold
Megan Blocker
What UX research maturity looks like and how we get there [Advancing Research Community Workshop Series] (Videoconference)
2023 • Advancing Research Community
Matt Bernius
Learnings from Applying Trauma-Informed Principles to the Research Process
2022 • Advancing Research 2022
Gold
Louis Rosenfeld
Founder’s Welcome
2022 • Design in Product 2022
Gold
Terry Buckman
Wargaming (An Introduction) (Videoconference)
2023 • Enterprise Community
Catt Small
Moving from Execution to Strategy as a Designer
2022 • Design in Product 2022
Gold

More Videos

"When teams lose a sense of agency, they act strange, cutting corners and racing to get something out the door."

Standardizing Product Merits for Leaders, Designers, and Everyone

June 15, 2018

Simon Wardley

"I realized I was just making it up despite the company’s growth."

Simon Wardley

Maps and Topographical Intelligence (Videoconference)

January 31, 2019

Sandra Camacho

"White supremacy is the belief system of the superiority of whiteness that can embed itself in algorithms."

Sandra Camacho

Creating More Bias-Proof Designs

January 22, 2025

Darian Davis

"Reviewing decision points creates a source of truth and makes it harder to forget or scrap earlier agreements."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Fisayo Osilaja

"Timely engagement helps stakeholders make faster, more informed decisions that lead to better business and product outcomes."

Fisayo Osilaja

[Demo] The AI edge: From researcher to strategist

June 4, 2024

Uday Gajendar

"There is a common thread in craft: dignity, purpose, utility, and beauty."

Uday Gajendar

The Wicked Craft of Enterprise UX

May 13, 2015

Davis Neable

"There is no design team responsible for the overarching experience across the entire company."

Davis Neable Guy Segal

How to Drive a Design Project When you Don’t Have a Design Team

June 10, 2021

Eniola Oluwole

"If you come with a big idea, they’ll try to dial you back to the smallest iota you can test first."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Aurobinda Pradhan

"The operational glue that binds strategy, execution, and measurement in design is often missing, and that’s critical for scaling."

Aurobinda Pradhan Shashank Deshpande

Introduction to Collaborative DesignOps using Cubyts

September 9, 2022