Rosenverse
[Demo] Stress-testing GenAI in user research synthesis

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis

Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Michelle Bejian Lotia
Rolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
2023 • Advancing Research 2023
Gold
Jason Mesut
Shaping design, designers and teams
2018 • DesignOps Summit 2018
Gold
Sarah Sgarlato Pierini
From Passion to Execution: A Story of Evolving Research Maturity at LinkedIn
2022 • DesignOps Summit 2022
Gold
Mike Oren
Improving Democratized Research with CustomGPTs and Gems
2026 • Rosenfeld Community
Lisanne Norman
Why I Left Research
2023 • Advancing Research 2023
Gold
Phil Gilbert
A Consistent Culture of Design
2015 • Enterprise UX 2015
Gold
Sylvie Abookire
A Civic Designer's Guide to Mindful Conflict Navigation
2022 • Civic Design 2022
Gold
Juhan Sonin
Design Now! The Agenda for Action
2025 • Rosenfeld Community
Abbey Smalley
Scaling UX Past the Size of Your Team
2024 • Enterprise Experience 2020
Gold
Bria Alexander
Opening Remarks
2022 • Civic Design 2022
Gold
Dr. Jamika D. Burge
How UX researchers can partner with (and not be replaced by) AI [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
JP Allen
Navigating the UX Tool Landscape
2021 • Advancing Research 2021
Gold
Harry Max
Failure Friday #5: Lessons from a SaaS Design Failure
2025 • Rosenfeld Community
Miles Orkin
Creativity and Culture
2018 • DesignOps Summit 2018
Gold
Kate Kalcevich
Integrating Accessibility in DesignOps
2024 • DesignOps Summit 2024
Gold
Connect with the Advancing Research Community
2026 • Advancing Research 2026
Gold

More Videos

Carl Turner

"People on the project were reporting back to management in their silo, not talking to one another."

Carl Turner

You Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project

March 28, 2023

Karen Pascoe

"If you have somebody on your team for three years, for five years, that’s amazing from a longevity perspective given enterprise legacy and complexity."

Karen Pascoe

Developing Experience Teams and Talent in the Enterprise

June 8, 2016

Jilanna Wilson

"Sending small care packages to remote team members goes a long way in showing we care and building connection."

Jilanna Wilson

Distributed Design Operations Management

October 23, 2019

Shawna Hein

"A government designer aims to reduce UX variation while a contractor may optimize only their single product, creating potential conflicts."

Shawna Hein Kevin Hoffman

Create a Cohesive Civic Design Practice Across Agency, Vendors, and Contracts

November 17, 2022

Barb Spanton

"You need to find your own mattress—a grounding purpose or phrase—that you can rely on when progress feels hopeless."

Barb Spanton

Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'

June 10, 2022

Briana Thomas

"We had no recipe, no formula, just vibes."

Briana Thomas

The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams

September 24, 2024

Gina Mendolia

"In large complex systems, the dots are scattered across silos and time zones, making progress feel elusive."

Gina Mendolia

Therapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems

December 3, 2024

Lija Hogan

"We need to become data ethicists and curators, not just data vacuum cleaners, to prioritize quality data."

Lija Hogan Milan Mijatovic Sam Proulx Louis Rosenfeld

Three Years Out: Perspectives on the Near-Term Future of User Research

March 15, 2024

John Calhoun

"With our new operating model, partners are more confident funding UX ops because of increased clarity and focus."

John Calhoun Rachel Posman

Two Sides of the DesignOps Coin: Teams Ops and Product Ops

January 8, 2024