[Demo] Stress-testing GenAI in user research synthesis
Summary
- Practical insights into integrating AI with traditional research methodologies to expedite the research process
- An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios
- Critical examination of AI's role in data processing and how it compares with human analysis
- Strategic considerations for service designers when employing AI to support rapid user research
- Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes
Key Insights
-
•
Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.
-
•
AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.
-
•
A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.
-
•
ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.
-
•
AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.
-
•
Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.
-
•
Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.
-
•
AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.
-
•
Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.
-
•
Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.
Notable Quotes
"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."
"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."
"I approached the GPT as if it were my human colleague, asking very open questions."
"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."
"This is a hero moment because it accelerates and simplifies exploration of clustering options."
"Text generation is the biggest superpower of ChatGPT in research synthesis right now."
"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."
"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."
"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."
"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."
Or choose a question:
More Videos
"We moved from Evernote to spreadsheets, to a giant affinity board in Mirror, then to UserVoice, and finally to idiomatic software with AI."
Shipra KayanHow we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
September 30, 2021
"Most categories are fuzzy; there’s no clear boundary between what’s in or out."
Peter MorvilleThe Architecture of Understanding
May 13, 2015
"Only one internal hire on the UX team was a huge misopportunity for us."
Renee ReidBecoming a ResearchH.E.R (Highly Enterprise Ready)
June 3, 2019
"Our bodies profoundly influence the way that we think about ourselves, others, the world, and everything in it."
Dane DeSutterKeeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
March 26, 2024
"Involving people with disabilities from the start reveals impact and usability issues that automated testing simply cannot catch."
Sam ProulxOnline Shopping: Designing an Accessible Experience
October 3, 2023
"Younger visitors are nearly twice as likely to engage with digital chat or help but are less satisfied afterward."
Andrew Custage Michael MallettThe Digital Journey: Research on Consumer Frustration and Loyalty
March 29, 2023
"Facilitation is part setting the stage and part performing on it."
Shelby SwitzerMaking Space for Community Knowledge-sharing in a Distributed World
December 10, 2021
"Right now, there’s a fear of being replaced by AI, and while a lot of that’s overblown, researchers have a legitimate reason to feel uncomfortable."
Llewyn PaineCoexisting with AI: A practical guide for researchers to navigate tools, ethics, and integration
March 11, 2025
"The only thing any of us can control is how quickly we learn and then act based on that learning."
Robin BeersNavigating organizational systems: Rethinking researcher’s role in driving change
March 13, 2025
