Rosenverse
Designing AI-first products on top of a rapidly evolving technology

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Designing AI-first products on top of a rapidly evolving technology

Gold
Tuesday, June 10, 2025 • Designing with AI 2025

This video is featured in the AI and UX playlist.

Share the love for this talk
Designing AI-first products on top of a rapidly evolving technology
Speakers: Peter Van Dijck
Link:

Summary

This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.

Key Insights

  • AI models evolve faster than previous tech revolutions, requiring adaptive product development.

  • Effective AI products use evals—automated tests on real data—to continuously measure quality.

  • Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.

  • AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.

  • Human factors like worker anxiety about job security are critical to address in AI adoption.

  • Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.

  • Prompt engineering skills grow best through extensive real use rather than frozen techniques.

  • Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.

  • Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.

  • User testing of AI products still requires old-school observation combined with analytics and evals.

Notable Quotes

"You can’t have good AI without good information architecture."

"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."

"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."

"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."

"Agents are models using tools in the loop, calling normal software functions as part of their process."

"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."

"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."

"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."

"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."

"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."

Ask the Rosenbot
Rebecca Gimenez
Work in Progress: Service Design at Airbnb
2024 • Advancing Service Design 2024
Gold
Etienne Fang
Power of Insights: Why sharing is better than silos with Uber’s Insights Platform
2019 • Advancing Research Community
John Taschek
Making People the X-Factor in the Enterprise
2018 • Enterprise Experience 2018
Gold
Florence Okoye
AfroFuturism and UX Research
2023 • Advancing Research 2023
Gold
Jemma Ahmed
Convergent Research Techniques in Customer Journey Mapping
2020 • Advancing Research 2020
Gold
Soma Ghosh
What emerging methods are advancing UX research [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Meghan Hellstern
The Next 100 Years of Civic Design: How Might We Better Rise to Meet the Challenges of Today and Tomorrow?
2021 • Civic Design 2021
Gold
Stephanie Wade
Building and Sustaining Design in Government
2021 • Civic Design 2021
Gold
Michaela Mora
Advanced Concept Testing Approaches To Guide Product Development and Business Decisions
2022 • Advancing Research 2022
Gold
Dr. Jamika D. Burge
The Future of Research: Bridging the Gaps
2021 • Advancing Research Community
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Ian Swinson
Designing and Driving UX Careers
2016 • Enterprise UX 2016
Gold
Megan Campos
What Did I Miss? The Hidden Costs of Deprioritizing Diversity in User Research
2021 • Advancing Research 2021
Gold
Sam Yen
Driving Organizational Change Through Design? Do more of this and less of that
2017 • Enterprise Experience 2017
Gold
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Melinda Belcher
Insider preview of Enterprise Experience 2020
2020 • Enterprise Community

More Videos

Cennydd Bowles

"Design permeates everything as a lens through which you see the world; philosophy is just a different shaped lens."

Cennydd Bowles

Exit Interview #2: Rediscovering the ethical heart of design

November 6, 2025

Katie Hansen

"The Homeowner Playbook helped align teams across product, marketing, and operations by giving everyone a shared research fact understanding."

Katie Hansen

Finding the unknown in the known: Harnessing meta-analysis and literature review

March 12, 2025

Sam Proulx

"Disability isn’t just a small percentage; one in five people live with a disability, and it’s an identity we all take on at some point."

Sam Proulx

Accessibility: An Opportunity to Innovate

March 9, 2022

Alexandra Schmidt

"No one is born knowing how to facilitate; it takes training, observation, and practice."

Alexandra Schmidt

Enterprise UX Playbook

December 1, 2022

Caroline Jarrett

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Caroline Jarrett

Garbage in, garbage out? Measuring error rates to get ready for AI

January 8, 2026

Meghan Bausone

"Paradigm shifts depend on how a problem is described and who controls the narratives and beliefs."

Meghan Bausone

Systems Thinking and Design Innovation: Working with Leverage Points in Rural Maternal Health Systems

April 17, 2026

Shelby Switzer

"Silence is productive—resist the temptation to fill it immediately after asking a question."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021

Bria Alexander

"Working on a different word than empower: People already have power, we have to respect it."

Bria Alexander Ariel Kennan Charlotte Lee Sarah Brooks Emily Lessard Gordon Ross Joanne Dong

Reflect and Chart Forward

December 10, 2021

Smitha Papolu

"We use the RICE method—reach, impact, confidence divided by effort—to prioritize customer requests in our sprints."

Smitha Papolu Nova Wehman-Brown Melissa Schmidt Adam Menter

Theme 3 Discussion

June 4, 2019