Integrating generative AI into enterprise products: A case study from dscout
Summary
Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.
Key Insights
-
•
Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.
-
•
Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.
-
•
Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.
-
•
Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.
-
•
Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.
-
•
Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.
-
•
The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.
-
•
Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.
-
•
Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.
-
•
Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.
Notable Quotes
"Building on AI means giving up control, which is tough but necessary."
"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."
"The product is no longer fixed; after the first interaction, it diverges user by user."
"We’re not studying products anymore; we’re studying experiences in ecosystems."
"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."
"Users are building mental models of how to use AI while they’re using it."
"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."
"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."
"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."
"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."
Or choose a question:
More Videos
"I like to engage my stakeholders as co-creators and co-discoverers so their ideas and concerns get reflected in recommendations."
Carl TurnerYou Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
March 28, 2023
"If there’s a performance gap, don’t sit on it; it doesn’t age well and hurts the team, employee, and business."
Karen PascoeDeveloping Experience Teams and Talent in the Enterprise
June 8, 2016
"Emojis and GIFs have become effective tools for remote teams to add expression and prevent miscommunication."
Jilanna WilsonDistributed Design Operations Management
October 23, 2019
"We’ve seen self-service tool usage grow from hundreds to millions of veterans served monthly after launching a unified VA platform."
Shawna Hein Kevin HoffmanCreate a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts
November 17, 2022
"Working in healthcare means every tiny moment in a patient’s experience can either uphold or erode their dignity and sense of control."
Barb SpantonDoing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
June 10, 2022
"We decided not to chase shiny objects because we wanted to stay true to our mission."
Briana ThomasThe Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
September 24, 2024
"Service designers connect the dots, which represent elements, teams, processes, policies, and customer needs that all must align."
Gina MendoliaTherapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
December 3, 2024
"Being fluent in multiple data languages and understanding behavioral and attitudinal data will be essential."
Lija Hogan Milan Mijatovic Sam Proulx Louis RosenfeldThree Years Out: Perspectives on the Near-Term Future of User Research
March 15, 2024
"Team ops goes wide to support 300-plus UX designers globally; product ops goes deep supporting 40-plus product design teams."
John Calhoun Rachel PosmanTwo Sides of the DesignOps Coin: Teams Ops and Product Ops
January 8, 2024