Rosenverse
[Demo] Stress-testing GenAI in user research synthesis

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis

Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Janelle Estes
UX Research Trends
2021 • Advancing Research Community
Marjorie Stainback
Transforming Strategic Research Capacity through Democratization
2019 • DesignOps Summit 2019
Gold
Edward Cupps
The Principal Path: Journeying from Management to Individual Contributor
2021 • Design at Scale 2021
Gold
Briana Thomas
The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
2024 • DesignOps Summit 2024
Gold
Jacqui Frey
Flow and Superfluidity for Design Orgs
2018 • DesignOps Summit 2018
Gold
Josh Clark
Sentient Design: New Postures for AI-Mediated Experiences (2nd of 3 seminars)
2025 • Rosenfeld Community
Roy Opata Olende
How Zapier Uses ‘All Hands Research’ to Increase Exposure to Users
2020 • Advancing Research Community
Laurent Christoph
Scale the impact of DesignOps in 3D: Diligence, Decision, Discipline
2025 • DesignOps Community
Bria Alexander
Opening Remarks
2022 • Civic Design 2022
Gold
Sean Fitzell
Craft of User Research: Building Out Jobs to be Done Maps
2021 • Advancing Research 2021
Gold
Saara Kamppari-Miller
Theme Three Intro
2022 • DesignOps Summit 2022
Gold
Simon Wardley
Maps and Topographical Intelligence
2019 • Enterprise Community
Sam Yen
Driving Organizational Change Through Design? Do more of this and less of that
2017 • Enterprise Experience 2017
Gold
Bria Alexander
Opening Remarks
2022 • DesignOps Summit 2022
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Amelia Cole
Data-Prompted Interviews
2021 • QuantQual Interest Group

More Videos

Jerome “Axle” Brown

"Everyone wants to see themselves reflected in the research questions and findings."

Jerome “Axle” Brown

How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon

March 11, 2021

Phil Gilbert

"The Austin studio’s walls can be put up and taken down by a 90-pound person on a Home Depot shopping cart."

Phil Gilbert

A Consistent Culture of Design

May 14, 2015

Kate Towsey

"Participants themselves need to be real humans at the end of the day to get unique insights."

Kate Towsey Basel Fakhoury Oren Friedman Graham Gardner

Participant Recruitment and Management Tools

March 12, 2026

Sohit Karol

"There hasn’t been a better time to be a user researcher, especially in machine learning enabled experiences."

Sohit Karol

Designing Delightful Listening Experiences: Mixed Methods Research in the Age of Machine Learning

March 31, 2020

Christopher Taylor Edwards

"Psychological safety comes from having hard conversations upfront and requires demonstration from leadership."

Christopher Taylor Edwards Valerie Roske

Design as a Team Practice, A Practical Guide to Cross-functional Collaboration

September 30, 2021

Sheryl Cababa

"Incentives influence behavior strongly, and lack of awareness about them leads to decisions misaligned with desired impact."

Sheryl Cababa

Expanding Your Design Lens with Systems Thinking

February 23, 2023

Francesca Barrientos, PhD

"Business is just people who may be working with design but don’t quite get how design helps them."

Francesca Barrientos, PhD

You Need Your Own Definition of Design Maturity

June 8, 2022

Allison Sanders

"I’ve onboarded 80% of the design and CX department, and it really shapes their trajectory and connection to culture."

Allison Sanders

Operating with Purpose

January 8, 2024

Saara Kamppari-Miller

"People were encouraging each other to score themselves higher and see their own potential."

Saara Kamppari-Miller

"Prototype" vs "Prototype"--Breaking Down and Rebuilding Our Understanding of What We Do

October 24, 2019