Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Case Study] Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Case Study] Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Speakers: Weidan Li
Link:

Summary

Following the emergence of Generative AI as a potential revolution in the UX field, a great deal of AI-driven tools arose to enhance the efficiency of UX research, including data analysis. Qualitative data analysis is a process that conventionally relies on human intelligence to discern patterns, establish connections, and derive actionable insights and frameworks. Many studies have involved comparing the quality of qualitative analyses generated by humans with those produced by AI language models like ChatGPT (Hamilton et al., 2023). Despite the undeniable appeal of automation and speed, there is ongoing debate about AI’s ability to replace human intelligence in qualitative analysis, which may be unlikely at this moment. Then the question is: To what extent can AI contribute to qualitative data analysis? In this case study, I delved into the thematic analysis and post-analysis stage, i.e. synthesizing insights into a framework. Framework, in this context, refers to a conceptual structure that illustrates the components of a human experience and how the components interconnect and operate within the structure. It is a concise model that encapsulates the entirety of research insights. The topic of my case study is "trust relationships between job seekers and hirers in the marketplace,, aligning with the business focus of my company. From my secondary research, I found that, ChatGPT needed multiple rounds of training using diverse prompts to conduct precise and comprehensive thematic analysis. ChatGPT can execute fine-quality thematic analysis under the help of right prompts, yet it falls short in replacing human intelligence for synthesizing insights and crafting frameworks for engaging narratives. Its limitation lies in lacking the depth of contextual understanding within a company, such as understanding what’s missing from the company’s mainstream discourse to create a human-centered story based on data analysis. To craft a framework that conveys good storytelling and organizational impact, it requires the researcher's introspection into knowledge gaps in the specific organizational context. Thus, the best practice is to combine human interpretation and AI production. In my talk, I will demonstrate several principles to guide this practice. Takeaways We’ll cover principles of how to employ ChatGPT in qualitative analysis, specifically focusing on its application in synthesizing and crafting frameworks that convey compelling and insightful narratives: Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately. Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis

Key Insights

  • ChatGPT provides thorough and comprehensive qualitative summaries, often catching details humans might miss due to fatigue.

  • AI struggles to create appropriate hierarchical organizing themes, often equating basic and organizing theme levels.

  • ChatGPT’s thematic outputs lack experiential connection and the subjective 'eye' humans have during data synthesis.

  • Frameworks generated by ChatGPT tend to be siloed, lacking interconnectivity and coherent storytelling.

  • ChatGPT fails to generate a single, cohesive analogy for complex conceptual frameworks, often producing disjointed metaphors.

  • Human curiosity and first-person immersion in data drive better abstraction and insight creation than AI currently can.

  • Manual qualitative synthesis resembles a cinematic experience, while AI synthesis is like watching a recap video.

  • Using AI as an assistive tool to validate and triangulate human findings leads to better insights than fully relying on AI.

  • Prompt customizations and role-playing with ChatGPT have limited ability to improve the quality of generated frameworks.

  • ChatGPT excels at quantitative content analysis tasks, such as counting word frequencies in qualitative data.

Notable Quotes

"How good is ChatGPT at synthesizing qualitative data?"

"The problem with ChatGPT’s organizing themes is many are at the same detail level as basic themes."

"AI doesn’t have an eye engaged in the sense-making experience, only rapid summaries."

"Qualitative synthesis is not supposed to be as straightforward as how AI does it."

"Manual qualitative synthesis is like watching a movie in a cinema, AI synthesis is like watching a recap video on YouTube."

"Curiosity is the main drive in fulfilling cognitive needs that lead to new knowledge and understanding."

"ChatGPT significantly outperforms me in thoroughness when analyzing large qualitative data sets."

"We shouldn’t take ChatGPT’s answers as conclusions but as stimuli for new perspectives."

"Using AI to quickly summarize is helpful, but we should slow down when human intelligence is needed for sensemaking."

"Prompt tweaks and role changes didn’t fundamentally improve framework quality because the problem lies in how insights are generated."

Ask the Rosenbot
Billy Carlson
Principles of Team Wireframing
2023 • DesignOps Summit 2023
Gold
Laine Riley Prokay
Carving a Path for Early Career DesignOps Practitioners
2022 • DesignOps Summit 2022
Gold
Sheri Byrne-Haber
The Importance of Accessible Design Systems
2024 • DesignOps Summit 2020
Gold
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold
Jerome “Axle” Brown
How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon
2021 • Advancing Research 2021
Gold
Brianna Sylver
Lead With Purpose
2020 • Advancing Research 2020
Gold
Dave Malouf
Theme 3: Introduction and Provocation
2024 • DesignOps Summit 2020
Gold
Abby Covert
Panel: Collaboration Tools
2017 • DesignOps Summit 2017
Gold
George Hinchliffe
Delivering Amazing Experiences
2021 • Design at Scale 2021
Gold
John Calhoun
Bring your DesignOps Story to Life! The Definitive DesignOps Book Jam
2023 • DesignOps Summit 2023
Gold
This Game is Never Done: Design Leadership Techniques from the Video Game World
2017 • DesignOps Summit 2017
Gold
Uday Gajendar
The Rise of Meta-Design: A Starter Playbook (Videoconference)
2022 • Enterprise Community
Andy Barraclough
From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
2024 • DesignOps 2024
Gold
Jaskiran Kang
Why Community is Key to Professionalizing Design (Videoconference)
2022 • Civic Design Community
Ellie Krysl
Planned Right. Managed Right. Designed Right.
2023 • Enterprise UX 2023
Gold
Ana Ferreira
Designing Distributed: Leading Doist’s Fully Remote Design Team in Six Countries
2024 • DesignOps Summit 2020
Gold

More Videos

"Development requests were often pragmatic—shadowing, mentoring, and reviewing—not big training courses."

Shaping design, designers and teams

November 8, 2018

Sabrina Mach

"You cannot sleep on your job; the design operating model must continuously evolve and be communicated."

Sabrina Mach Nina Wainwright

How to Design Your Design Operating Model

September 29, 2021

Samuel Proulx

"If you think IE identity verification is necessary, ask why and if you really have to do it."

Samuel Proulx

Invisible barriers: Why accessible service design can’t be an afterthought

December 3, 2024

Saara Kamppari-Miller

"Meeting designers where they spend their time and integrating accessibility into their tools is the most effective way to drive adoption."

Saara Kamppari-Miller

DesignOps for Inclusive Design and Accessibility (Videoconference)

May 26, 2022

Briana Thomas

"Product design and development can often mirror what happens in a chef’s kitchen: lots of coordination and spinning plates behind the scenes."

Briana Thomas

The Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams

September 24, 2024

Adam Cutler

"I wish I could tell my 22-year-old self what to stop doing and what to embrace to be a better designer."

Adam Cutler Karen Pascoe Ian Swinson Susan Worthman

Discussion

June 8, 2016

"We wanted to move fast in the right direction, not just going anywhere for the sake of going there."

Operationalizing DesignOps

November 7, 2018

Leah Buley

"I don’t even use the word UX or user experience with clients if I can help it. Jargon is a barrier."

Leah Buley Joe Natoli

Ask Me Anything with Leah Buley and Joe Natoli, co-authors of The User Experience Team of One (2nd edition)

October 8, 2024

Nathan Shedroff

"If your budgets don’t connect to strategy goals, maybe those budget items shouldn’t exist."

Nathan Shedroff

Double Your Mileage: Use Your Research Strategically

March 31, 2020