This video is featured in the AI and UX playlist.
Summary
Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.
Key Insights
-
•
LLM-based products are no longer fixed, creating unique, evolving user experiences rather than identical ones.
-
•
Traditional usability testing is challenged by AI's unpredictability, requiring longitudinal and repeated user interactions.
-
•
Designers and researchers must embrace imperfection and failure as inherent parts of AI product development.
-
•
Wizard of Oz testing—using humans to simulate AI—remains a valuable method to rapidly test AI-driven interactions before full tech build.
-
•
Model drift can cause AI outputs to become inconsistent, requiring strategies like turn capping or memory integration to manage.
-
•
Users must build mental models of AI systems as they interact with them, demanding new onboarding and education approaches.
-
•
LLMs introduce stochastic behavior, so the same user command may yield different outputs across sessions.
-
•
Trust and relationship building between users and AI agents are complex and require thoughtful, context-sensitive design.
-
•
AI can free human users from mundane tasks, enabling more critical thinking, creativity, and authentic self-expression.
-
•
Cross-functional alignment and multi-dimensional prioritization remain essential but must accommodate AI’s uncertain and evolving nature.
Notable Quotes
"Building on AI means giving up control."
"If you’re right all the time, you’re probably not taking big enough swings."
"The product is no longer the dependent variable; it diverges with each user interaction."
"We’re not studying products anymore, we’re studying experiences in ecosystems."
"Failure is inevitable in AI product development, so embrace it early and fail fast."
"You have to test relationships with AI longitudinally because relationships evolve over time."
"Many users have never had a human assistant, so digital assistants require building new mental models."
"The folks who design these technologies don’t really know how they work — it’s a black box."
"Latency and asynchronous conversation are surprisingly important and hard to replicate in bots."
"Creativity, accountability, and authenticity will be the new markers of humanity in an AI world."
Dig deeper—ask the Rosenbot:
More Videos
"The Product React uses reaction time to capture the user’s gut feeling that otherwise remains hidden."
Daniela Magaña Flores Ariane Rahn Jeff Ephraim BanderAhead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
March 10, 2022
"AI tools can do the entire process of qualitative analysis at the click of a button."
Savina HawkinsHarnessing AI in UXR: Practical Strategies for Positive Impact
March 26, 2024
"Each species lost is a thread untethered from the web of life that supports us all."
Alex Hurworth Bonnie John Fahd Arshad Antoine MarinDesigning a Contact Tracing App for Universal Access
October 23, 2020
"The goal is to transform raw data into wisdom — knowing what to do with the knowledge gathered."
Matt DuignanHITS, Microsoft's internal human insight system: From research library to living body of knowledge (Videoconference)
July 16, 2019
"Designers need to speak truth to power and engage their creative abilities to materially change how and what is produced - Diego Rodriguez."
Craig VillamorDesign Systems for Ethical Design (Videoconference)
January 26, 2023
"The goal isn't to get people to understand design or research; the goal is to get human-centered design into products that ship."
Ovetta SampsonTurning UX Passion into Real Product Influence
June 7, 2023
"The problem isn’t ever fully defined. The numbers give you the what, but they don’t give you the why."
Rima Campbell Amrit S BhachuIncrease Productivity and Drive Business Impact
September 24, 2024
"Making friends is about 75% of the game in my job because we solve problems together."
Amy Gawronski ZuccaroAdvice for DesignOps Employee #1
September 29, 2021
"Systems of value determine whose knowledge we increase and whose knowledge we diminish."
Victor UdoewaBeyond Methods and Diversity: The Roots of Inclusion
March 26, 2024
