Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters
Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO IT department is only 3% of the company but manages IT for 12,000 IBMers; the CIO design team is just 1% of that group.

  • Over half of IBM’s 385,000 employees have been hired within five years, introducing ongoing challenges in onboarding and experience design.

  • 30% of IBMers work remotely or offsite, necessitating resilient, intuitive digital tools without onsite support.

  • The team uses an IT pyramid of pain inspired by Maslow’s hierarchy to ensure foundational tech and network reliability before focusing on tasks.

  • Continuous measurement includes tracking NPS, goal completion, ease of use, and user comments linked directly to employee IDs for rich insights.

  • IBM shifted from infrequent large surveys to daily embedded Voice of the Employee forms capturing immediate feedback on 4,000 submissions/month.

  • The design ops team integrates research, UX, content, and coaching to foster collaboration and prevent silos.

  • A predictive AI model helps identify when employees should upgrade their hardware before critical failures occur, leading IBM to offer on-demand new machines.

  • Transparency in metrics drives improvement and accountability, with NPS trending to be incorporated into employee evaluations starting next year.

  • Kristen emphasizes embracing chaos and iteration in design ops rather than forcing rigid processes, highlighting the team's evolution from phase one simplicity to complex maturity.

Notable Quotes

"This small but mighty CIO design team is literally 1% of the 3% IT population."

"Over half of IBMers have been here less than five years, adding complexity to designing their experience."

"The state of IT is a daily reflection of what IBM thinks and feels about its employees."

"You can’t just focus on the task level, you have to consider the entire ecosystem and its layers."

"People take out a pen when the CIO says I have a design user experience and research function reporting directly to me."

"We’re here to make work better, and the coaching tribe is here to make teams better."

"We moved from large infrequent surveys to daily Voice of the Employee forms capturing real-time user sentiment."

"Every minute spent struggling with an IT system is a minute not delivering value to IBM and our clients."

"Don’t try to control or force data to say what you want. The goal is for data to help make the right decision."

"Measurement matters. Even from a small circle, you can make a big impact."

Ask the Rosenbot
Rich Mironov
How Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
2022 • Design in Product 2022
Gold
Theresa Slate
Why Changing Hearts & Minds Doesn’t Work When Promoting DE&I Efforts, but Checklists Do
2023 • DesignOps Summit 2023
Gold
Husani Oakley
Bias Towards Action: Building Teams that Build Work
2018 • Enterprise Experience 2018
Gold
Wendy Johansson
Design at Scale: Behind the Scenes (Videoconference)
2021 • Enterprise Community
Cennydd Bowles
Day 1 Panel
2024 • Designing with AI 2024
Gold
Christopher Taylor Edwards
Design as a Team Practice, A Practical Guide to Cross-functional Collaboration
2021 • DesignOps Summit 2021
Gold
Julie Gitlin
Design as an Agent of Digital Transformation at JPMC
2021 • Design at Scale 2021
Gold
Jonathan Fairman
Integrating generative AI into enterprise products: A case study from dscout
2024 • Designing with AI 2024
Gold
Sarah Alvarado
How to make UX research leadership more effective [Advancing Research Community Workshop Series] (Videoconference)
2023 • Advancing Research Community
Uday Gajendar
The Wicked Craft of Enterprise UX
2015 • Enterprise UX 2015
Gold
Craig Brookes
"Just Make it Look Good" and Other Ways We're Misunderstood
2021 • Design at Scale 2021
Gold
Dr. Karl Jeffries
The Science of Creativity for DesignOps
2024 • DesignOps Summit 2020
Gold
Tatyana Mamut
Opening Keynote: Breaking Conway's Law--or How to Work Differently and Not Ship Your Org Chart
2019 • Enterprise Experience 2019
Gold
Yulya Besplemennova
[Demo] Stress-testing GenAI in user research synthesis
2024 • Designing with AI 2024
Gold
Saara Kamppari-Miller
Inclusive Design is DesignOps
2021 • DesignOps Summit 2021
Gold
Dan Hill
Designing for the infrastructures of everyday life
2024 • Designing with AI 2024
Gold

More Videos

"The important thing was the conversation around the visual maps, not the size or scale of areas."

Shaping design, designers and teams

November 8, 2018

Sabrina Mach

"You cannot sleep on your job; the design operating model must continuously evolve and be communicated."

Sabrina Mach Nina Wainwright

How to Design Your Design Operating Model

September 29, 2021

Samuel Proulx

"Disability isn’t a problem with a person; it’s something that happens when we fail to enable them."

Samuel Proulx

Invisible barriers: Why accessible service design can’t be an afterthought

December 3, 2024

Saara Kamppari-Miller

"It’s very overwhelming for one person to own accessibility alone; you have to build a coalition of passionate allies."

Saara Kamppari-Miller

DesignOps for Inclusive Design and Accessibility (Videoconference)

May 26, 2022

Briana Thomas

"Product design and development can often mirror what happens in a chef’s kitchen: lots of coordination and spinning plates behind the scenes."

Briana Thomas

The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams

September 24, 2024

Adam Cutler

"The 1500 are designers, we don't distinguish strictly between UX or visual or industrial—they all bring design to the org."

Adam Cutler Karen Pascoe Ian Swinson Susan Worthman

Discussion

June 8, 2016

"Designers can focus on designing and developers can focus on developing — people are getting paid to do their jobs now."

Operationalizing DesignOps

November 7, 2018

Leah Buley

"If you are spinning and can’t get your team aligned, the issue is you have not talked to real customers recently."

Leah Buley Joe Natoli

Ask Me Anything with Leah Buley and Joe Natoli, co-authors of The User Experience Team of One (2nd edition)

October 8, 2024

Nathan Shedroff

"Design researchers know customers better than almost anyone else in the organization, yet they are rarely invited into strategy processes."

Nathan Shedroff

Double Your Mileage: Use Your Research Strategically

March 31, 2020