Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Summary
Enthusiasm for AI tools, especially large language models like ChatGPT, is everywhere, but what does it actually look like to deliver large-scale user-facing experiences using these tools in a production environment? Clearly they're powerful, but what do they need to make them work reliably and at scale? In this session, Sarah provides a perspective on some of the information architecture and user experience infrastructure organizations need to effectively leverage AI. She also shares three AI experiences currently live on Microsoft Learn: An interactive assistant that helps users post high-quality questions to a community forum A tool that dynamically creates learning plans based on goals the user shares A training assistant that clarifies, defines, and guides learners while they study Through lessons learned from shipping these experiences over the last two years, UXers, IAs, and PMs will come away with a better sense of what they might need to make these hyped-up technologies work in real life.
Key Insights
-
•
Most AI applications no longer require building foundation models from scratch; the focus is now on application development and integration.
-
•
Single, all-purpose chatbots (everything chatbots) are insufficient because they handle high ambiguity and diverse, often complex tasks poorly.
-
•
Sarah introduces the ambiguity footprint as a framework to measure AI application complexity and risks across several axes such as task complexity, context, interface, prompt openness, and sensitivity.
-
•
AI features that support simple, complimentary user tasks, rather than critical or complex ones, are easier and safer to build and scale.
-
•
Visible AI interfaces, like chatbots, set clearer user expectations but introduce more ambiguity and management overhead compared to invisible AI (e.g., keyboard optimizations).
-
•
Prompt engineering plays a crucial role in defining the boundaries of AI output, from very open-ended to highly restricted scopes.
-
•
Retrieval Augmented Generation (RAG) helps manage up-to-date context by dynamically querying relevant data chunks rather than using static corpus.
-
•
Evaluating AI outputs rigorously is essential but often underprioritized; without clear quality metrics, teams end up relying on subjective or anecdotal assessments.
-
•
Data ethics and distributed AI implementations can create blind spots, limiting feedback loops necessary for continuous AI model improvement.
-
•
Incrementally building AI applications with smaller ambiguity footprints helps organizations develop expertise and controls before tackling more complex, open-ended AI products.
Notable Quotes
"You’re not doing IA, but you’re always doing it."
"An everything chat bot is almost certainly not how you’re going to build it; realistically you’re building three apps in a trench coat."
"AI is ambiguous at best because we’re fully in the realm of probabilistic rather than deterministic programming."
"The more complex the task, the less likely it is to be successful with current AI."
"A task where AI adds a little something is honestly easier to get right than one where it’s absolutely critical."
"Visible AI interfaces introduce another place where you can add ambiguity."
"Retrieval Augmented Generation lets you supply specific relevant information to the model dynamically rather than everything at once."
"Evaluation might be the most important part of your entire development effort and is often the hardest to do well."
"You can’t just eyeball results and call it good; AI applications are expensive and complex and require systematic evaluation."
"Never build or buy an everything chat bot again; start with less ambiguous, targeted AI experiences."
Or choose a question:
More Videos
"If we wanted to scale and continue to grow, we were going to need something to help us."
Taylor Jennings Joe Nelson Alex KnollRepository Retrospective: Learnings from Introducing a Central Place for UX Research
March 9, 2022
"Expectation describes how things ought to be and reinforces the distinction between the present and what is not yet here."
Nicole AleongFuture Orientations to Everyday Life: Futures Anthropology as a Methodology
March 26, 2024
"The innovation studio bakes in strategic vision, leadership, funding, and ownership."
Jeff GothelfInnovation Studios: the Engines of Enterprise Experimentation
May 14, 2015
"By designing products through the lens of edge cases like disabilities first, they often become better products for everyone."
Saara Kamppari-MillerDesignOps for Inclusive Design and Accessibility
May 26, 2022
"There’s a lot of pressure on ops people and managers to make everything perfect all the time for staff."
Tess DixonC'mon Get Happy
September 29, 2021
"Soft skills often are a multiplier on your hard skills—they’re just as valuable, if not more so."
Liam ThurstonWhy Your Design Team Is Quitting, And How To Fix It
June 10, 2022
"Communication matters—sending an email to an executive with spelling mistakes can ruin your respect."
Ian SwinsonDesigning and Driving UX Careers
June 8, 2016
"Ratios always start on the premise of spreading people too thin, which is not a great way to get success."
Leisa ReicheltOpening Keynote: Operating in Context
November 7, 2018
"Without a design ops function, teams make their own flavors that dilute brand and accessibility."
Rachael Greene Alison DavisBuilding a Design Ops Practice that Really Works (Most of the Time)
October 2, 2025
Latest Books All books
Dig deeper with the Rosenbot
What are the main limitations of using AI for moderating user research interviews?
What are effective behaviors that design operations teams should focus on to deliver value and build trust?
How can play help overcome the ‘messy middle’ between research insights and decision-making in organizations?