Disrupting generative AI products with just-in-time consumer insights
Summary
Join Katie Johnson, Head of Consumer Insights at PanasonicWELL, to explore the future of developing products and services atop Generative AI technology. This talk will cover how to think about experimenting with users today for use cases that don’t yet exist at scale, and how to bring insightful findings to leadership to make strategic decisions quickly. Katie will shed light on methods and strategies she’s employed in building products with cutting-edge technology throughout her career in agency life, blockchain, Google’s 0 to 1 environment in Assistant, and now at PanasonicWELL.
Key Insights
-
•
Building AI products requires giving up control over user experience as outputs vary uniquely per user.
-
•
Traditional usability testing is insufficient for AI products; longitudinal testing with repeated sessions is critical.
-
•
Human-in-the-loop 'Wizard of Oz' testing effectively simulates AI to rapidly test assumptions before building.
-
•
Model drift in LLMs can cause unpredictable or inappropriate responses, requiring mechanisms like turn capping.
-
•
Users form evolving relationships with AI assistants, which need careful design and testing over time.
-
•
Many product orthodoxies based on fixed experiences expire with generative AI and must be reinvented.
-
•
Failure early and often is essential in AI product development due to its inherent unpredictability.
-
•
Users may lack mental models for AI assistants, making onboarding and user education a bigger challenge.
-
•
Asynchronous, multi-threaded conversations are natural for humans but difficult for bots to handle accurately.
-
•
AI can automate synthesis tasks, enabling humans to focus on creativity, accountability, and authenticity.
Notable Quotes
"Building on AI and generative AI means giving up control."
"If the folks that design this tech don’t know how it works, it’s okay if you don’t know either."
"Alignment inevitably means changing minds."
"Everyone’s gonna be wrong. If you’re right all the time, you’re probably not taking big enough swings."
"With LLMs, the product itself is no longer a dependent variable; experiences diverge per user."
"We are studying experiences in ecosystems, not just products anymore."
"Failing early is critical because you can’t control the model."
"The promise of an LLM assistant requires users to build new mental models while using it."
"Turn capping helps reset the model to prevent drift but introduces new problems around memory."
"Creativity, accountability, and authenticity are going to be the new markers of humanity."
Or choose a question:
More Videos
"Our technical leads started attending other teams’ meetings, helping break down integration barriers."
Carl TurnerYou Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
March 28, 2023
"Consumers are clamoring for payment to be embedded in their devices because payment can be a pain point in digital commerce."
Karen PascoeDeveloping Experience Teams and Talent in the Enterprise
June 8, 2016
"We share the inconvenience of meeting times across time zones, alternating so no one team always bears it."
Jilanna WilsonDistributed Design Operations Management
October 23, 2019
"We’ve seen self-service tool usage grow from hundreds to millions of veterans served monthly after launching a unified VA platform."
Shawna Hein Kevin HoffmanCreate a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts
November 17, 2022
"Don’t kill grandma is our mantra reminding us the stakes of the tiny design details we face every day."
Barb SpantonDoing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
June 10, 2022
"What our users see is the perfect plate, but there is so much unseen work behind it."
Briana ThomasThe Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
September 24, 2024
"In large complex systems, the dots are scattered across silos and time zones, making progress feel elusive."
Gina MendoliaTherapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
December 3, 2024
"Future UX includes designing for user discovery in conversational and augmented interfaces, not just clicks."
Lija Hogan Milan Mijatovic Sam Proulx Louis RosenfeldThree Years Out: Perspectives on the Near-Term Future of User Research
March 15, 2024
"Our design ops team was a step behind the growth of our design team, solving the wrong problems at the wrong scale."
John Calhoun Rachel PosmanTwo Sides of the DesignOps Coin: Teams Ops and Product Ops
January 8, 2024
