Rosenverse
Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?

Friday, February 18, 2022 • QuantQual Interest Group
Share the love for this talk
Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?
Speakers: Brian T. O’Neill , Maria Cipollone , Luis Colin , Manuel Dahm and Mike Oren
Link:

Summary

Join us for a different type of Quant vs. Qual discussion: instead of discussing how data science and quantitative research methods can power UX research and design, we’re going to talk about designing enterprise data products and tools that put ML and analytics into the hands of users. Does this call for new, different, or modified approaches to UX research and design? Or do these technologies have nothing to do with how we approach design for data products?

The session’s host will be Brian T. O’Neill who is also the host of the Experiencing Data podcast and founder of Designing for Analytics, an independent consultancy that helps data products leaders use design-driven innovation to deliver better ML and analytics user experiences. In this session, we’ll be sharing some rapid [slides-optional] anecdotes and stories from the attendees and then open up the conversation to everyone.

We hope to get perspectives from both enterprise data teams doing “internal” data analytics or ML/AI solutions development as well as software/tech companies as well who may offer data-related platform tools, intelligence/SAAS products, BI/decision support solutions, etc. Slots are open to both experienced UX practitioners as well as data science / analytics / technical participants who may have participated in design or UX work with colleagues. Please share! If folks are too quiet in the session, you may be subject to a drum or tambourine solo from Brian. Nobody has all of this “figured out yet” and experiments and trials are welcome.

Key Insights

  • •

    UX researchers in ML-heavy environments often face steep domain knowledge gaps requiring strong interview facilitation to bridge communication.

  • •

    Designing for AI data products demands a shift from generic user interfaces to tools tailored for expert users who need explainability over polish.

  • •

    Cross-functional collaboration thrives by creating shared spaces that respect differing expertise while focusing on a unifying product goal.

  • •

    Decision culture focusing on how decisions are made is more useful than a vague data culture mindset when designing AI/ML products.

  • •

    Besides customers and stakeholders, data labelers who curate training data form a critical, often overlooked human part of the ML ecosystem.

  • •

    Model interpretability can be more important than raw accuracy because users must trust a system to adopt it and generate value.

  • •

    Prototyping data products requires believable data and can be facilitated by testing edge cases like false positives and trust thresholds early.

  • •

    Incremental learning loops involving user feedback can improve models over time and should be designed into AI products.

  • •

    Contextualizing data model outputs with business rules and user environment (e.g., seasonal factors) increases relevance and trust.

  • •

    No-code data science tools and scenario building help teams quickly experiment with data products despite the usual slow model training cycles.

Notable Quotes

"Smoke comes out of my ears when I'm talking to some of the more advanced applications in machine learning."

"Design is about creating that shared space where we look at problems through different lenses."

"Decision culture is a better lens than data culture for thinking about what decisions we're trying to facilitate with AI."

"Not everyone using machine learning is your user; think also about the people labeling the data feeding these models."

"If nobody uses this because they don’t trust it, it doesn’t matter how accurate the model is."

"You need to prototype with real or believable data so the data doesn’t change the context of use."

"Some things that look like blockers, like domain expertise gaps, actually force you to become a better interviewer."

"We’re not always trying to use machine learning everywhere—sometimes the answer is to ignore it."

"Users want to understand why this happened, what will happen, and how can we make it happen—explainability is critical."

"The presenting problem might be a dashboard with a score, but the real need is deciding what to do with that information."

Ask the Rosenbot
Briana Thomas
The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
2024 • DesignOps Summit 2024
Gold
Milan Guenther
The $212 billion ‘so what?’: unlocking impact in development cooperation
2025 • Advancing Service Design 2025
Gold
Janelle Estes
UX Research Trends
2021 • Advancing Research Community
Mackenzie Cockram
Integrating Qualitative and Quantitative Research from Discovery to Live
2022 • QuantQual Interest Group
Phil Hesketh
Designing Accessible Research Workflows
2021 • DesignOps Summit 2021
Gold
Andy Warr
Under My (Research) Umbrella: The Benefits and Challenges of Building a Unified Insights Function
2024 • Advancing Research 2024
Gold
Eniola Oluwole
Lessons From the DesignOps Journey of the World's Largest Travel Site
2019 • DesignOps Summit 2019
Gold
Brenna Fallon
Learning Over Outcomes
2019 • DesignOps Summit 2019
Gold
Amy Paris
Delivering Equity: Government Services for All Ages, Languages, Sexual Orientations, and Gender Identities
2021 • Civic Design 2021
Gold
Ron Bronson
Design, Consequences & Everyday Life
2022 • Civic Design 2022
Gold
Scott Jensen
Short Take #2: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Jake Burghardt
Stop wasting research: Create new value with insight summaries
2025 • Rosenfeld Community
Lisanne Norman
Why I Left Research
2023 • Advancing Research 2023
Gold
Iulia Cornigeanu
QuantQual Book Club: Small Data
2024 • QuantQual Interest Group
Kaitlin Tasker
Fast and Fearless Inclusive Research
2023 • Advancing Research 2023
Gold
Brad Orego
Bringing Customer Research to More Internal Teams
2022 • Advancing Research 2022
Gold

More Videos

Jon Fukuda

"Showing where you’re using research more visibly creates a virtuous cycle of adoption and impact."

Jon Fukuda Jake Burghardt Jose Coronado Natalie Dunbar Denise Tilles

All the Ops: Successful cross-functional collaboration

September 10, 2025

Alana Washington

"We’ve got Tim Gilligan, Meredith Blackbrand, and Bria Alexander on a panel about the future of design ops."

Alana Washington

Theme 3 Intro

October 1, 2021

Andrew Michael

"Marketing is an essential collaborator because they help package and communicate insights organization-wide."

Andrew Michael

Building a Product Insights Team

March 10, 2022

Nalini P. Kotamraju

"The leadership response in business often has to be: we will make the best decision possible given what we know."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021

Prayag Narula

"We shouldn't let the world of computer science make us forget our own power and what we bring to the table."

Prayag Narula Rida Qadri

HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer

March 28, 2023

Yunyan Li

"Post-implementation reviews help catch bugs or style issues before customers see them, maintaining high quality."

Yunyan Li Anna Le Jen Kim

UX Best Practices

June 11, 2021

Shipra Kayan

"Going from 100 to 200 people, suddenly product managers started hearing from a dozen people a week with conflicting requests."

Shipra Kayan

How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up

September 30, 2021

Dorelle Rabinowitz

"Bring people along on a journey. Lead people towards your side by focusing on the future state."

Dorelle Rabinowitz

The Magic Word is Trust

June 15, 2018

Sarah Brooks

"It’s been a real highlight of the year to work with this incredibly distinguished group of speakers."

Sarah Brooks

Theme 3 Intro

December 10, 2021