Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?
Friday, February 18, 2022 • QuantQual Interest Group (Rosenfeld Community)
Share the love for this talk
Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?
Speakers: Brian T. O’Neill , Maria Cipollone , Luis Colin , Manuel Dahm and Mike Oren
Link:

Summary

Join us for a different type of Quant vs. Qual discussion: instead of discussing how data science and quantitative research methods can power UX research and design, we’re going to talk about designing enterprise data products and tools that put ML and analytics into the hands of users. Does this call for new, different, or modified approaches to UX research and design? Or do these technologies have nothing to do with how we approach design for data products? The session’s host will be Brian T. O’Neill who is also the host of the Experiencing Data podcast and founder of Designing for Analytics, an independent consultancy that helps data products leaders use design-driven innovation to deliver better ML and analytics user experiences. In this session, we’ll be sharing some rapid [slides-optional] anecdotes and stories from the attendees and then open up the conversation to everyone. We hope to get perspectives from both enterprise data teams doing “internal” data analytics or ML/AI solutions development as well as software/tech companies as well who may offer data-related platform tools, intelligence/SAAS products, BI/decision support solutions, etc. Slots are open to both experienced UX practitioners as well as data science / analytics / technical participants who may have participated in design or UX work with colleagues. Please share! If folks are too quiet in the session, you may be subject to a drum or tambourine solo from Brian. Nobody has all of this “figured out yet” and experiments and trials are welcome.

Key Insights

  • UX researchers working with ML and data science teams often lack domain expertise, requiring strong facilitation and interpretive interview skills.

  • Machine learning product users are usually small specialized expert groups, making statistical rigor and sampling difficult.

  • Trust and interpretability in ML models can be more important than peak accuracy for business adoption.

  • Collaborative shared spaces or cross-functional teams focused on common goals bridge gaps between design, engineering, and data science.

  • Designers help translate data science outputs into actionable, contextual decision support for end users.

  • Prototyping ML products requires believable, realistic data and testing boundaries like false positives early to build trust.

  • Decision culture (focusing on which decisions to support) is a more useful framing than data culture in enterprise AI/ML products.

  • The ecosystem around ML includes not only users and business stakeholders but also data labelers who impact outcomes.

  • The integration of business rules with ML models enhances contextual relevance and facilitates user trust.

  • No-code and rapid data science tools are emerging to help speed experimentation but do not fully replace model development demands.

Notable Quotes

"When you work with machine learning engineers doing advanced techniques, you’re really out of the realm of your knowledge."

"Most data science and analytics teams do not have designers or user experience people unless they are software native companies."

"If nobody uses this because they don’t trust it, it doesn’t matter. You just rehearsed without a concert."

"We need to think more about decision culture instead of data culture — what decisions are we trying to make?"

"Sometimes, the answer is ignore machine learning here. It is not the right tool for everything."

"Designers partnering with data scientists leads to smarter, more adaptable interfaces that actually get used."

"We’re talking to really small groups — sometimes 8 to 10 ML engineers on a particular domain — so sampling and rigor are tough."

"Creating that shared space where design, engineering, and data science work together is key to success."

"The human algorithms that people use today should be understood and incorporated into ML models where possible."

"You have to test the boundaries of false positives, false negatives, and surprising positives to see if people trust your model."

Ask the Rosenbot
Noel Lamb
Cultivating Business Partnerships to Grow Research Ops (Videoconference)
2022 • Advancing Research Community
Katie Johnson
Disrupting generative AI products with just-in-time consumer insights
2024 • Designing with AI 2024
Gold
Sahibzada Mayed
The Politics of Radical Research: A Manifesto
2023 • Advancing Research 2023
Gold
Mackenzie Guinon
M.C. Escher’s UX Research Career Ladder
2022 • Advancing Research 2022
Gold
Tatyana Mamut
Opening Keynote: Breaking Conway's Law--or How to Work Differently and Not Ship Your Org Chart
2019 • Enterprise Experience 2019
Gold
The Unspoken Complexity of “Self-Care” with Deanna Zandt
2022 • Civic Design Community
Jorge Arango
[Demo] How to re-categorize content at scale using LLMs
2024 • Designing with AI 2024
Gold
Group Activity: Making Sense of DesignOps
2017 • DesignOps Summit 2017
Gold
Josh Clark
Sentient Design: New Design Patterns for New Experiences (3rd of 3 seminars)
2025 • Rosenfeld Community
Farid Sabitov
Theme Four Intro
2022 • DesignOps Summit 2022
Gold
Andrew Michael
Building a Product Insights Team
2022 • Advancing Research 2022
Gold
Amelia Cole
Data-Prompted Interviews
2021 • QuantQual Interest Group (Rosenfeld Community)
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Erin Malone
Understanding the past to prepare for the future
2024 • Rosenfeld Community
Chelsea Mauldin
Let's Talk About Money
2022 • Civic Design 2022
Gold
Meredith Black
Scaling Design Culture
2017 • DesignOps Summit 2017
Gold

More Videos

"If design didn’t move the needle, we wouldn’t be having this conversation."

Standardizing Product Merits for Leaders, Designers, and Everyone

June 15, 2018

Simon Wardley

"Building a map is like learning to play chess—you have to see the board to decide your move."

Simon Wardley

Maps and Topographical Intelligence (Videoconference)

January 31, 2019

Sandra Camacho

"Many smartwatch designs assume male hands as default, excluding smaller wrists and diverse body types."

Sandra Camacho

Creating More Bias-Proof Designs

January 22, 2025

Darian Davis

"Reviewing decision points creates a source of truth and makes it harder to forget or scrap earlier agreements."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Fisayo Osilaja

"Having conversations with stakeholders before sharing AI-generated recommendations validates their legitimacy and relevance."

Fisayo Osilaja

[Demo] The AI edge: From researcher to strategist

June 4, 2024

Uday Gajendar

"Working with CEOs like Mark Templeton is like Dancing with the Stars — an interpretive dance of translating fuzzy ideas."

Uday Gajendar

The Wicked Craft of Enterprise UX

May 13, 2015

Davis Neable

"No official kickoff was held, which meant shifting expectations and lack of alignment."

Davis Neable Guy Segal

How to Drive a Design Project When you Don’t Have a Design Team

June 10, 2021

Eniola Oluwole

"TripAdvisor is really nine major business units each thinking like sub companies instead of one end-to-end user experience."

Eniola Oluwole

Lessons From the DesignOps Journey of the World's Largest Travel Site

October 24, 2019

Aurobinda Pradhan

"Our two-way Jira integration allows you to link design activities directly with epics, user stories, and tasks."

Aurobinda Pradhan Shashank Deshpande

Introduction to Collaborative DesignOps using Cubyts

September 9, 2022