Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Integrating generative AI into enterprise products: A case study from dscout

Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
Integrating generative AI into enterprise products: A case study from dscout
Speakers: Jonathan Fairman and Kevin Johnson
Link:

Summary

Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.

Key Insights

  • Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.

  • Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.

  • Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.

  • Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.

  • Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.

  • Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.

  • The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.

  • Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.

  • Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.

  • Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.

Notable Quotes

"Building on AI means giving up control, which is tough but necessary."

"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."

"The product is no longer fixed; after the first interaction, it diverges user by user."

"We’re not studying products anymore; we’re studying experiences in ecosystems."

"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."

"Users are building mental models of how to use AI while they’re using it."

"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."

"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."

"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."

"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."

Ask the Rosenbot
Verónica Urzúa
The B-side of the Research Impact
2021 • Advancing Research 2021
Gold
Abbey Smalley
Scaling UX Past the Size of Your Team
2024 • Enterprise Experience 2020
Gold
Dave Malouf
Closing Keynote: Amplify. Not Optimize.
2019 • DesignOps Summit 2019
Gold
Ricardo Martins
Unlocking the power of advanced quantitative methods
2025 • Advancing Research 2025
Gold
Sara Conklin
Exit Interview: 20 Years of Tech, One Very Big Bet, and a Lot of Heat Pumps
2026 • Rosenfeld Community
Peter Van Dijck
Building impactful AI products for design and product leaders, Part 3: Understand AI architectures: RAG, Agents, Oh My!
2025 • Rosenfeld Community
Bill Scott
Lean Engineering: Engineering for Learning and Experimentation in the Enterprise
2015 • Enterprise UX 2015
Gold
Dana Bishop
2022: The Year UX Demonstrates its Business Impact
2022 • Advancing Research 2022
Gold
Josina Vink
Navigating the pitfalls of systems thinking in service design
2024 • Advancing Service Design 2024
Gold
Michelle Bejian Lotia
Rolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
2023 • Advancing Research 2023
Gold
Uday Gajendar
Leading through the long tail of trauma
2022 • Advancing Research Community
Jon Fukuda
Design Planning and Management Support
2023 • DesignOps Summit 2023
Gold
Mike Oren
Improving Democratized Research with CustomGPTs and Gems
2026 • Rosenfeld Community
Sam Proulx
SUS: A System Unusable for Twenty Percent of the Population
2021 • DesignOps Summit 2021
Gold
Sarah Brooks
Fireside chat with Sarah Brooks and Jen Pahlka
2021 • Civic Design Community
Justin Entzminger
Risk and Reward: How to Diversify the Field of Civic Innovators and Designers
2022 • Civic Design 2022
Gold

More Videos

Xenia Adjoubei

"The use of dovetail helps organize diverse information without losing the author’s voice or connection to original data."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Tamara Kartoziia

"Service design stops being just a support function; it becomes the strategic driver that transforms organizational performance."

Tamara Kartoziia

Think global, adapt local: how service design accelerated B2B market entry by 6 months

November 20, 2025

Jay Bustamante

"AI tech debt has compounding interest to it."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Nicole Aleong

"Destiny is the belief that a certain future outcome is predetermined regardless of the path."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Nick Lewis

"Using generative AI for design risks losing the human creativity that makes design special."

Nick Lewis

Designing and building low-carbon websites independently

November 18, 2025

Vicky Teinaki

"I recently observed a director of engineering talking to a content strategist about what they were both seeing because they were both in the actual physical room at the same time."

Vicky Teinaki Michele Marut Tim Parmee

Short Take #3: UX/Product Lessons from Your Industry Peers

December 6, 2022

Laura Weiss

"Conflict occurs when something important to us feels threatened or disrespected."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Joerg Beringer

"You get a lot of different research outputs in a matter of minutes from your scope input."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

June 10, 2025

Shelby Switzer

"Why be here if you’re not going to be here."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021