Key Insights

• •

  LLM-based products are no longer fixed, creating unique, evolving user experiences rather than identical ones.

• •

  Traditional usability testing is challenged by AI's unpredictability, requiring longitudinal and repeated user interactions.

• •

  Designers and researchers must embrace imperfection and failure as inherent parts of AI product development.

• •

  Wizard of Oz testing—using humans to simulate AI—remains a valuable method to rapidly test AI-driven interactions before full tech build.

• •

  Model drift can cause AI outputs to become inconsistent, requiring strategies like turn capping or memory integration to manage.

• •

  Users must build mental models of AI systems as they interact with them, demanding new onboarding and education approaches.

• •

  LLMs introduce stochastic behavior, so the same user command may yield different outputs across sessions.

• •

  Trust and relationship building between users and AI agents are complex and require thoughtful, context-sensitive design.

• •

  AI can free human users from mundane tasks, enabling more critical thinking, creativity, and authentic self-expression.

• •

  Cross-functional alignment and multi-dimensional prioritization remain essential but must accommodate AI’s uncertain and evolving nature.

Notable Quotes