Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI can significantly speed up text generation and help brainstorm clustering parameters in research synthesis.

  • AI struggles to assign specific interviewees to user archetypes without detailed human guidance.

  • The best use of AI involves a semi-automated process combining AI brainstorming with human refinement.

  • Automated visuals generated by DALL·E often fail to accurately represent complex user concepts.

  • AI behaves like an 'alien intern,' lacking common sense and understanding of research processes.

  • Providing AI with explicit details like participant names improves clustering and assignment accuracy.

  • AI can efficiently retrieve specific quotes from transcripts when given open-ended queries.

  • Human experience is crucial to evaluate and guide AI outputs, preventing misleading or irrelevant results.

  • AI reduces cognitive overload and accelerates synthesis during tight deadlines and solo work.

  • Rapid AI-assisted research risks sacrificing depth and nuance of human insights for speed.

Notable Quotes

"Gene AI can really help to speed up some certain synthesis moments, especially those connected to textual content generation and navigation."

"AI behaves kind of like an alien intern because it’s not capable of making sense of some things which are easy for us."

"I had to adapt and follow a semi-automated process where I would brainstorm a lot of things with AI, then define many things by myself."

"The biggest superpower of ChatGPT right now is generating textual descriptions along the given format when it understands it and the relevance."

"Visuals are not working that well with DALL·E in this context, despite multiple instructions to remove cables or irrelevant elements."

"AI can provide multiple inputs that stimulate human thinking even if many are generic and need human filtering."

"You need to guide AI a lot and do handholding, even reminding it simple things like how many people are in the transcript."

"It’s better to have a human, but if you don’t, AI can be a partner in the process to speed things up."

"The value is not so much about the results but about helping manage cognitive overload and speed up the process under tight deadlines."

"AI making the process five times faster is worrying if it means losing the depth of human insights for speed and efficiency."

Ask the Rosenbot
Anna Poznyakov
Get The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact
2021 • Advancing Research 2021
Gold
Lija Hogan
Practical Principles of Inclusive Research
2023 • Advancing Research 2023
Gold
Discussion
2017 • Enterprise Experience 2017
Gold
Kim Holt
A Salesforce Panel Discussion on Values-Driven DesignOps
2022 • DesignOps Summit 2022
Gold
Asia Hoe
Partnering with Product: A Journey from Junior to Senior Design
2023 • Design in Product 2023
Gold
Aditi Ruiz
A PM State of Mind: Empathy Mapping Your Product Manager, Pt. 1
2022 • Design in Product 2022
Gold
Uday Gajendar
The Rise of Meta-Design: A Starter Playbook (Videoconference)
2022 • Enterprise Community
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Phil Hesketh
Designing Accessible Research Workflows
2021 • DesignOps Summit 2021
Gold
Shipra Kayan
How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
2021 • DesignOps Summit 2021
Gold
Gregg Bernstein
Opportunistic Research with Gregg Bernstein (Videoconference)
2019 • Advancing Research Community
Matt Duignan
Atomizing Research: Trend or Trap
2020 • Advancing Research 2020
Gold
Frances Yllana
The Big Question about Impact: A Panel Discussion
2024 • DesignOps 2024
Gold
Kristin Skinner
Opening Keynote: Org Design for Design Orgs
2017 • DesignOps Summit 2017
Gold
Saara Kamppari-Miller
Theme Three Intro
2023 • DesignOps Summit 2023
Gold
Jim Kalbach
Jobs To Be Done (Videoconference)
2021 • Enterprise Community

More Videos

"Many designers feel overwhelmed by hundreds of expected skills, which creates paralysis about where to focus."

Shaping design, designers and teams

November 8, 2018

Sabrina Mach

"Design needs to be aligned first; if design cannot agree amongst itself, how can the bigger organisation understand it."

Sabrina Mach Nina Wainwright

How to Design Your Design Operating Model

September 29, 2021

Samuel Proulx

"The later you consider accessibility, the more complicated and costly the solutions become."

Samuel Proulx

Invisible barriers: Why accessible service design can’t be an afterthought

December 3, 2024

Saara Kamppari-Miller

"By designing products through the lens of edge cases like disabilities first, they can often become better products for everyone."

Saara Kamppari-Miller

DesignOps for Inclusive Design and Accessibility (Videoconference)

May 26, 2022

Briana Thomas

"Dana and Davis weren’t seen on stage at Adobe Max, but you felt their impact throughout the experience."

Briana Thomas

The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams

September 24, 2024

Adam Cutler

"Don't be cool, be good—work hard at managing your teams because they need you to nail it."

Adam Cutler Karen Pascoe Ian Swinson Susan Worthman

Discussion

June 8, 2016

"I’m hoping that once we figure this out, I can kind of go back to my old goal and break those rules."

Operationalizing DesignOps

November 7, 2018

Leah Buley

"If you are spinning and can’t get your team aligned, the issue is you have not talked to real customers recently."

Leah Buley Joe Natoli

Ask Me Anything with Leah Buley and Joe Natoli, co-authors of The User Experience Team of One (2nd edition)

October 8, 2024

Nathan Shedroff

"You need to learn the vocabulary, tools, and processes your peers know and understand where they are inadequate."

Nathan Shedroff

Double Your Mileage: Use Your Research Strategically

March 31, 2020