Disrupting generative AI products with just-in-time consumer insights
Summary
Join Katie Johnson, Head of Consumer Insights at PanasonicWELL, to explore the future of developing products and services atop Generative AI technology. This talk will cover how to think about experimenting with users today for use cases that don’t yet exist at scale, and how to bring insightful findings to leadership to make strategic decisions quickly. Katie will shed light on methods and strategies she’s employed in building products with cutting-edge technology throughout her career in agency life, blockchain, Google’s 0 to 1 environment in Assistant, and now at PanasonicWELL.
Key Insights
-
•
Building AI products requires giving up control over user experience as outputs vary uniquely per user.
-
•
Traditional usability testing is insufficient for AI products; longitudinal testing with repeated sessions is critical.
-
•
Human-in-the-loop 'Wizard of Oz' testing effectively simulates AI to rapidly test assumptions before building.
-
•
Model drift in LLMs can cause unpredictable or inappropriate responses, requiring mechanisms like turn capping.
-
•
Users form evolving relationships with AI assistants, which need careful design and testing over time.
-
•
Many product orthodoxies based on fixed experiences expire with generative AI and must be reinvented.
-
•
Failure early and often is essential in AI product development due to its inherent unpredictability.
-
•
Users may lack mental models for AI assistants, making onboarding and user education a bigger challenge.
-
•
Asynchronous, multi-threaded conversations are natural for humans but difficult for bots to handle accurately.
-
•
AI can automate synthesis tasks, enabling humans to focus on creativity, accountability, and authenticity.
Notable Quotes
"Building on AI and generative AI means giving up control."
"If the folks that design this tech don’t know how it works, it’s okay if you don’t know either."
"Alignment inevitably means changing minds."
"Everyone’s gonna be wrong. If you’re right all the time, you’re probably not taking big enough swings."
"With LLMs, the product itself is no longer a dependent variable; experiences diverge per user."
"We are studying experiences in ecosystems, not just products anymore."
"Failing early is critical because you can’t control the model."
"The promise of an LLM assistant requires users to build new mental models while using it."
"Turn capping helps reset the model to prevent drift but introduces new problems around memory."
"Creativity, accountability, and authenticity are going to be the new markers of humanity."
Or choose a question:
More Videos
"We were interested in supporting networks of self-organization and mutual aid formed between refugees and host communities."
Xenia Adjoubei Sean BruceEmpowering Communities Through the Researcher in Residence Program
March 29, 2023
"Notice period sometimes can be more than three months for some roles, so parallel activities are essential."
Tamara KartoziiaThink global, adapt local: how service design accelerated B2B market entry by 6 months
November 20, 2025
"You can’t put the toothpaste back in the tube once biased AI harms your brand or users."
Jay BustamanteNavigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation
October 2, 2023
"Anticipation involves preparatory actions to manage uncertainty about future conditions."
Nicole AleongFuture Orientations to Everyday Life: Futures Anthropology as a Methodology
March 26, 2024
"The Climate Dictionary was built in a couple hours over a weekend, focused on function over form."
Nick LewisDesigning and building low-carbon websites independently
November 18, 2025
"I recently observed a director of engineering talking to a content strategist about what they were both seeing because they were both in the actual physical room at the same time."
Vicky Teinaki Michele Marut Tim ParmeeShort Take #3: UX/Product Lessons from Your Industry Peers
December 6, 2022
"Conflict actually strengthens relationships if we choose to engage with it."
Laura WeissTurn Down the Heat: 3 Ways to Handle Conflict in the Moment
November 20, 2024
"Most product teams don't really have user research outputs or they come too late and disconnected from design."
Joerg Beringer Thomas GeisScaling User Research with AI: Continuous Discovery of User Needs in Minutes
June 10, 2025
"Zoom fatigue is real, people are not necessarily keen to go into another Zoom meeting even if they think it’s valuable."
Shelby SwitzerMaking Space for Community Knowledge-sharing in a Distributed World
December 10, 2021
