Integrating generative AI into enterprise products: A case study from dscout
Summary
Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.
Key Insights
-
•
Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.
-
•
Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.
-
•
Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.
-
•
Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.
-
•
Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.
-
•
Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.
-
•
The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.
-
•
Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.
-
•
Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.
-
•
Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.
Notable Quotes
"Building on AI means giving up control, which is tough but necessary."
"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."
"The product is no longer fixed; after the first interaction, it diverges user by user."
"We’re not studying products anymore; we’re studying experiences in ecosystems."
"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."
"Users are building mental models of how to use AI while they’re using it."
"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."
"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."
"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."
"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."
Or choose a question:
More Videos
"Leadership is enabling people to become empowered to achieve what they are set to achieve."
Anna Avrekh Dr. John Pagonis Klara Pelcl Sina SchreiberExpert Panel: Leading in and with Research
March 10, 2022
"The UX field's expansion leads to pushback—people ask why researchers talk about areas owned by others like OCM."
Taylor KlassmanShaping the Next Era of UX Research: Collaborative Forum
March 11, 2025
"We have to figure out the strategic, unique value to our clients, stakeholders, and customers."
Dr. Jamika D. Burge Steve Portigal Alba Villamil Sam LadnerThe Future of Research: Bridging the Gaps
July 29, 2021
"Participant experience is everything and it starts with building trust."
Marisa BernsteinIt Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing
December 9, 2021
"The goal is that design systems become the way that we do work, just like when you do Agile, you don’t talk about it—you just do it."
Dan Mall“Ask Me Anything” with Dan Mall, Author of Upcoming Rosenfeld Title, Design that Scales
October 2, 2023
"What am I personally doing to contribute to the dynamics that stand in the way of better futures?"
Chris GeisonTheme 1 Intro
March 9, 2022
"It’s worth explaining your new communication approach to stakeholders to save their time and foster discussion."
Bruce GillespieLearning from journalism: Balancing impactful communication with compassionate storytelling
March 13, 2025
"If you put garbage in, garbage comes out – AI reflects the biases in its training data."
Nancy DouyonWe'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex
June 15, 2018
"We had conversations with donors over dinners and asked about their vision for innovation and future impact—it’s about asking the right questions."
Liz EbengoThe Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward
December 4, 2024