Designing AI-first products on top of a rapidly evolving technology
This video is featured in the AI and UX playlist.
Summary
This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"Design permeates everything as a lens through which you see the world; philosophy is just a different shaped lens."
Cennydd BowlesExit Interview #2: Rediscovering the ethical heart of design
November 6, 2025
"The Homeowner Playbook helped align teams across product, marketing, and operations by giving everyone a shared research fact understanding."
Katie HansenFinding the unknown in the known: Harnessing meta-analysis and literature review
March 12, 2025
"Disability isn’t just a small percentage; one in five people live with a disability, and it’s an identity we all take on at some point."
Sam ProulxAccessibility: An Opportunity to Innovate
March 9, 2022
"No one is born knowing how to facilitate; it takes training, observation, and practice."
Alexandra SchmidtEnterprise UX Playbook
December 1, 2022
"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."
Caroline JarrettGarbage in, garbage out? Measuring error rates to get ready for AI
January 8, 2026
"Paradigm shifts depend on how a problem is described and who controls the narratives and beliefs."
Meghan BausoneSystems Thinking and Design Innovation: Working with Leverage Points in Rural Maternal Health Systems
April 17, 2026
"Silence is productive—resist the temptation to fill it immediately after asking a question."
Shelby SwitzerMaking Space for Community Knowledge-sharing in a Distributed World
December 10, 2021
"Working on a different word than empower: People already have power, we have to respect it."
Bria Alexander Ariel Kennan Charlotte Lee Sarah Brooks Emily Lessard Gordon Ross Joanne DongReflect and Chart Forward
December 10, 2021
"We use the RICE method—reach, impact, confidence divided by effort—to prioritize customer requests in our sprints."
Smitha Papolu Nova Wehman-Brown Melissa Schmidt Adam MenterTheme 3 Discussion
June 4, 2019
