Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI can significantly speed up text generation and help brainstorm clustering parameters in research synthesis.

  • AI struggles to assign specific interviewees to user archetypes without detailed human guidance.

  • The best use of AI involves a semi-automated process combining AI brainstorming with human refinement.

  • Automated visuals generated by DALL·E often fail to accurately represent complex user concepts.

  • AI behaves like an 'alien intern,' lacking common sense and understanding of research processes.

  • Providing AI with explicit details like participant names improves clustering and assignment accuracy.

  • AI can efficiently retrieve specific quotes from transcripts when given open-ended queries.

  • Human experience is crucial to evaluate and guide AI outputs, preventing misleading or irrelevant results.

  • AI reduces cognitive overload and accelerates synthesis during tight deadlines and solo work.

  • Rapid AI-assisted research risks sacrificing depth and nuance of human insights for speed.

Notable Quotes

"Gene AI can really help to speed up some certain synthesis moments, especially those connected to textual content generation and navigation."

"AI behaves kind of like an alien intern because it’s not capable of making sense of some things which are easy for us."

"I had to adapt and follow a semi-automated process where I would brainstorm a lot of things with AI, then define many things by myself."

"The biggest superpower of ChatGPT right now is generating textual descriptions along the given format when it understands it and the relevance."

"Visuals are not working that well with DALL·E in this context, despite multiple instructions to remove cables or irrelevant elements."

"AI can provide multiple inputs that stimulate human thinking even if many are generic and need human filtering."

"You need to guide AI a lot and do handholding, even reminding it simple things like how many people are in the transcript."

"It’s better to have a human, but if you don’t, AI can be a partner in the process to speed things up."

"The value is not so much about the results but about helping manage cognitive overload and speed up the process under tight deadlines."

"AI making the process five times faster is worrying if it means losing the depth of human insights for speed and efficiency."

Michele Marut
Research Repositories Reconsidered (Videoconference)
2019 • DesignOps Community
Molly Fargotstein
Multipurpose Communication & UX Research Marketing (Videoconference)
2019 • DesignOps Community
Courtney Maya George
Scale Your Organization and Grow Your Designers
2022 • DesignOps Summit 2022
Gold
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps 2024
Gold
Jorge Arango
[Demo] How to re-categorize content at scale using LLMs
2024 • Designing with AI 2024
Gold
Billy Carlson
Tips to Utilize Wireframes to Tell an Effective Product Story
2023 • Enterprise UX 2023
Gold
Prayag Narula
Empowering Designers to do Good Research
2022 • Advancing Research 2022
Gold
Kim Holt
A Salesforce Panel Discussion on Values-Driven DesignOps
2022 • DesignOps Summit 2022
Gold
Uday Gajendar
The Wicked Craft of Enterprise UX
2015 • Enterprise UX 2015
Gold
Kim Lenox
Leading Distributed Global Teams (Videoconference)
2019 • Enterprise Community
Angelos Arnis
State of DesignOps: Learnings from the 2021 Global Report
2021 • DesignOps Summit 2021
Gold
Jon Fukuda
Design Planning and Management Support
2023 • DesignOps Summit 2023
Gold
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Carla Casariego
DesignOps in Wonderland
2019 • DesignOps Summit 2019
Gold
Chris Geison
What's Next for Research? (Videoconference)
2021 • Advancing Research Community
Maish Nichani
Sparking a Service Excellence Mindset at a Government Agency
2021 • Civic Design 2021
Gold

More Videos

Monty Hammontree

"Satya Nadella talks about empathy a lot; it’s at the center of everything he does authentically."

Monty Hammontree

The Future of UX Research (Videoconference)

December 3, 2020

Liz Ebengo

"Hope is not passive; it is proactive and must be facilitated carefully, especially where optimism is scarce."

Liz Ebengo

The Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward

December 4, 2024

Billy Carlson

"It’s really powerful to leave question marks in wireframes to open the conversation with your presentation audience."

Billy Carlson

Tips to Utilize Wireframes to Tell an Effective Product Story

June 6, 2023

Saara Kamppari-Miller

"The better question is who are we excluding when we talk about inclusive design."

Saara Kamppari-Miller

Inclusive Design is DesignOps

September 29, 2021

Ian Swinson

"Your career is the only design project you will ever own, so own it consciously and actively."

Ian Swinson

Designing and Driving UX Careers

June 8, 2016

Jonathon Colman

"We trashed our old content design competencies because content designers should hold themselves to the same standards as product designers."

Jonathon Colman

How to Maximize the Impact of Content Design

January 8, 2024

Maria Skaaden

"What users say they want and what they really need can be very different; you need to be careful about how you interpret feedback."

Maria Skaaden

Continuous Design: One eye on the horizon and the other on the next wave

November 8, 2018

Bryce Benton

"Python is great for writing scrapers, and these large language models are great for writing Python."

Bryce Benton

[Demo] AI-powered UX enhancement: Aligning GitHub documentation with USWDS at Austin Public Library

June 4, 2024

Marc Fonteijn

"Men in this report consistently report higher income than women, but don’t draw early conclusions — seniority levels and positions matter."

Marc Fonteijn

First Insights from the 2025 Service Design Salary(+) Report

December 4, 2024