Rosenverse
When AI Becomes the User’s Point Person—and Point of Failure

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

When AI Becomes the User’s Point Person—and Point of Failure

Thursday, August 7, 2025 • Rosenfeld Community
Share the love for this talk
When AI Becomes the User’s Point Person—and Point of Failure
Speakers: Heidi Trost
Link:

Summary

Imagine slipping on a sleek pair of smart glasses. Not only do you look sharp, the glasses capture everything you see, hear, and do. Your AI assistant—built into the glasses and synced to your email, social media accounts, health apps, and finances—manages your life. It’s tasked with paying bills, booking trips, replying to messages, even helping you swipe right.

Over time, you find yourself chitchatting with your AI assistant. You call him Charlie.

Now imagine you’re a threat actor. That trust between user and AI assistant? It’s your entry point.

If your product is powered by AI, you’re not just designing features—you’re designing an entire relationship. You’re designing Charlie. Let’s talk about where that goes wrong—and how to get it right.

Key Insights

  • •

    Users often do not understand why AI-powered systems request extensive personal data, increasing privacy risks.

  • •

    Trust in AI agents can become excessive, creating new vectors for manipulation by threat actors.

  • •

    Security issues typically occur beneath the surface until alerts disrupt the user experience, often causing frustration.

  • •

    Prompt injection attacks pose a novel threat where malicious inputs manipulate AI agents to access sensitive user data.

  • •

    Multimodal AI interfaces introduce complexity in security decisions, increasing chances for user errors.

  • •

    Secure by default settings reduce burden on users and improve overall protection without requiring user intervention.

  • •

    Cross-disciplinary collaboration between UX, security, product, legal, and compliance teams is crucial for safer AI design.

  • •

    Users need clear, contextual guidance during onboarding to make informed decisions about data sharing and security settings.

  • •

    Transparency about AI limitations and giving users the option to reverse AI actions are essential for building trust.

  • •

    Threat actors are likely to exploit growing AI access to personal data and automate vulnerabilities discovery.

Notable Quotes

"When a product is powered by AI, you're not just designing the features; you are designing an entire relationship."

"Charlie is like the most annoying coworker who constantly surfaces problems but never offers solutions to Alice."

"Threat actors probably know your system better than you do and are looking for any entry points to exploit."

"Alice often perceives Charlie as just another barrage of alerts filled with jargon she doesn't understand."

"Prompt injection attacks can trick AI agents into accessing private data like emails without the user realizing."

"People become incrementally more comfortable giving away data because they see the value AI provides."

"We need secure defaults that protect users out of the box without them having to figure it out."

"Alert fatigue is real; users can't be burdened with constant security decisions or they'll ignore them."

"Giving users the ability to reverse AI-driven actions is critical but currently underexplored."

"If Charlie has been tampered with, Alice needs a clear way to be alerted that she shouldn't trust it."

Ask the Rosenbot
Ivana Ng
Level Up Your Program with ProductOps
2024 • Enterprise Experience 2020
Gold
Michelle Morrison
Practice What You Preach
2024 • DesignOps Summit 2020
Gold
Kyria Stephens
Power to Heal: Civic Design in the Aftermath of Tragedy
2022 • Civic Design 2022
Gold
Bassel Deeb
Do More With Less: Equip and Lead Design Orgs Through Adversity
2023 • DesignOps Summit 2023
Gold
Landon Barnes
Are My Research Findings Actually Meaningful?
2022 • Advancing Research 2022
Gold
Llewyn Paine
Day 1 Using AI in UX with Impact
2025 • Designing with AI 2025
Gold
Dan Mall
“Ask Me Anything” with Dan Mall, Author of Upcoming Rosenfeld Title, Design that Scales
2023 • DesignOps Summit 2023
Gold
Jenny Price
From Tradition to Transformation: Unlocking Startup Agility in a Legacy Enterprise
2025 • DesignOps Summit 2025
Gold
Robin Beers
Panel: Excellence in Communicating Insights
2024 • Advancing Research 2024
Gold
Shipra Kayan
Make your research synthesis speedy and more collaborative using a canvas
2025 • Rosenfeld Community
Tristin Oldani
Turning awareness into action with Climate UX
2025 • Climate UX Interest Group
Mujtaba Hameed
Frameworks for Excellence: Using Visual Thinking and Communication to Elevate Your Research
2024 • Advancing Research 2024
Gold
Jon Temple
Panel: Stacks, Security, and Stakeholders: The Hidden Work of UXR Tool Procurement
2026 • Advancing Research 2026
Gold
Nicole Aleong
Future Orientations to Everyday Life: Futures Anthropology as a Methodology
2024 • Advancing Research 2024
Gold
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Mohammad Hossein Jarrahi
Contextuality problem: Exploring the Benefits of Qualitative and Quantitative Research
2023 • QuantQual Interest Group

More Videos

Dave Malouf

"Make a proof of concept before investing a lot more time and energy into a design system."

Dave Malouf Amy Thibodeau

Panel: Design Systems and Documentation

November 7, 2017

John Donmoyer

"Having tight feedback loops and directly controlling the end-to-end experience completely changed how excited I was to do design work."

John Donmoyer

Shipping your code generation experiments to production

June 11, 2025

Clara Kliman-Silver

"An AI-driven design tool should help me by lowering the redundant work, reducing tedious tasks and help me focus on the experience and strategy."

Clara Kliman-Silver

UX Futures: The Role of Artificial Intelligence in Design

June 7, 2023

Mark Templeton

"User experiences and customer experiences are feelings—we design those feelings more than features."

Mark Templeton

Creating a Legacy: the ultimate experience

June 9, 2017

Sarah Gallimore

"The mayor wanted to shrink the neighborly bonds that the city was known for by investing in a 2030 living Monument of collaboration and community."

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Sam Proulx

"Webflow is an ideal tool for prototypes with code because users can test in the actual browser environment."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

December 8, 2021

Anna Nguyen

"We realized that everyone kind of has a different definition of the voice of customer and that we needed to define our own to make it relevant for us."

Anna Nguyen Emily Brogan

Why Our Voice of the Customer is Better Than Yours

March 10, 2022

Anna Poznyakov

"How do we continuously provide value throughout the product development cycle without overindexing on just one type of research?"

Anna Poznyakov Richa Prajapati

Get The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact

March 12, 2021

Nalini P. Kotamraju

"Transparency and honesty are the only ways I handle difficult questions about compromises for the greater good."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021