Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Integrating generative AI into enterprise products: A case study from dscout

Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
Integrating generative AI into enterprise products: A case study from dscout
Speakers: Jonathan Fairman and Kevin Johnson
Link:

Summary

Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.

Key Insights

  • Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.

  • Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.

  • Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.

  • Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.

  • Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.

  • Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.

  • The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.

  • Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.

  • Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.

  • Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.

Notable Quotes

"Building on AI means giving up control, which is tough but necessary."

"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."

"The product is no longer fixed; after the first interaction, it diverges user by user."

"We’re not studying products anymore; we’re studying experiences in ecosystems."

"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."

"Users are building mental models of how to use AI while they’re using it."

"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."

"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."

"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."

"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."

Ask the Rosenbot
Wyatt Hayman
Global Research Panels
2020 • DesignOps Community
Zariah Cameron
Streamlining an Inclusive Design Practice
2023 • DesignOps Summit 2023
Gold
Cennydd Bowles
Responsible Design in Reality
2021 • Design at Scale 2021
Gold
Kit Unger
Theme 2: Discussion
2024 • Enterprise Experience 2020
Gold
Joe Meersman
Use AI to Drive Outcomes that Go Beyond the Design Sprint
2024 • DesignOps Summit 2024
Gold
Louis Rosenfeld
Coffee with Lou: Should You Write a (UX) Book?
2024 • Rosenfeld Community
Changying (Z) Zheng
Navigating Innovation with Integrity
2024 • DesignOps Summit 2024
Gold
Jorge Arango
Design as an Antidote to VUCA
2019 • Enterprise Community
Chris Hodowanec
Agile + User Experience: How to navigate the Agile landscape as an UX Practitioner
2022 • Civic Design 2022
Gold
Andrea Gallagher
The Problem Space
2019 • Advancing Research Community
Dave Hora
Advice for Establishing Research
2022 • Advancing Research Community
Peter Van Dijck
Building impactful AI products for design and product leaders, Part 3: Understand AI architectures: RAG, Agents, Oh My!
2025 • Rosenfeld Community
Sohit Karol
Designing Delightful Listening Experiences: Mixed Methods Research in the Age of Machine Learning
2020 • Advancing Research 2020
Gold
Bruce Gillespie
Learning from journalism: Balancing impactful communication with compassionate storytelling
2025 • Advancing Research 2025
Gold
Dave Hora
Research in the Face of Complexity: New Sensibility for New Situations
2025 • Rosenfeld Community
George Abraham
Design Systems To-Go: Introducing a Starter Design System, and Indigo.Design Overview (Part 1)
2021 • DesignOps Summit 2021
Gold

More Videos

Taylor Jennings

"Short-term learnings kind of roll into long-term ones as you iterate and build on historical findings."

Taylor Jennings Joe Nelson Alex Knoll

Repository Retrospective: Learnings from Introducing a Central Place for UX Research

March 9, 2022

Nicole Aleong

"There are over 7 billion futures of right now."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Jeff Gothelf

"These are all attempts by large companies to figure out new ways to deliver their products or services or to create new businesses outside of what they’re currently working on."

Jeff Gothelf

Innovation Studios: the Engines of Enterprise Experimentation

May 14, 2015

Saara Kamppari-Miller

"Accessibility ownership should never fall on just one person but be understood and shared across entire product teams."

Saara Kamppari-Miller

DesignOps for Inclusive Design and Accessibility

May 26, 2022

Tess Dixon

"Design ops folks meet the habit of catering so much to everybody’s increasingly trivial whims that it takes us away from the real meaty ops work."

Tess Dixon

C'mon Get Happy

September 29, 2021

Liam Thurston

"Cross-functional relationships often matter more to output than relationships within the immediate team."

Liam Thurston

Why Your Design Team Is Quitting, And How To Fix It

June 10, 2022

Ian Swinson

"Your career is a design project. It’s the only one you own, so own it like you would a product."

Ian Swinson

Designing and Driving UX Careers

June 8, 2016

Leisa Reichelt

"At DTA, distributed teams and lack of local diversity meant everyone was out in the field a lot to get diverse input."

Leisa Reichelt

Opening Keynote: Operating in Context

November 7, 2018

Rachael Greene

"Getting to know someone as a person first—the vibe check—is the first step to successful UX research collaboration."

Rachael Greene Alison Davis

Building a Design Ops Practice that Really Works (Most of the Time)

October 2, 2025