Rosenverse
Qualitative synthesis with ChatGPT: Better or worse than human intelligence?

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Qualitative synthesis with ChatGPT: Better or worse than human intelligence?

Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Speakers: Weidan Li
Link:

Summary

Following the emergence of Generative AI as a potential revolution in the UX field, a great deal of AI-driven tools arose to enhance the efficiency of UX research, including data analysis. Qualitative data analysis is a process that conventionally relies on human intelligence to discern patterns, establish connections, and derive actionable insights and frameworks. Many studies have involved comparing the quality of qualitative analyses generated by humans with those produced by AI language models like ChatGPT (Hamilton et al., 2023). Despite the undeniable appeal of automation and speed, there is ongoing debate about AI’s ability to replace human intelligence in qualitative analysis, which may be unlikely at this moment. Then the question is: To what extent can AI contribute to qualitative data analysis? In this case study, I delved into the thematic analysis and post-analysis stage, i.e. synthesizing insights into a framework. Framework, in this context, refers to a conceptual structure that illustrates the components of a human experience and how the components interconnect and operate within the structure. It is a concise model that encapsulates the entirety of research insights. The topic of my case study is "trust relationships between job seekers and hirers in the marketplace,, aligning with the business focus of my company. From my secondary research, I found that, ChatGPT needed multiple rounds of training using diverse prompts to conduct precise and comprehensive thematic analysis. ChatGPT can execute fine-quality thematic analysis under the help of right prompts, yet it falls short in replacing human intelligence for synthesizing insights and crafting frameworks for engaging narratives. Its limitation lies in lacking the depth of contextual understanding within a company, such as understanding what’s missing from the company’s mainstream discourse to create a human-centered story based on data analysis. To craft a framework that conveys good storytelling and organizational impact, it requires the researcher's introspection into knowledge gaps in the specific organizational context. Thus, the best practice is to combine human interpretation and AI production. In my talk, I will demonstrate several principles to guide this practice. Takeaways We’ll cover principles of how to employ ChatGPT in qualitative analysis, specifically focusing on its application in synthesizing and crafting frameworks that convey compelling and insightful narratives: Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately. Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis

Key Insights

  • ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.

  • ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.

  • Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.

  • AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.

  • Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.

  • Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.

  • Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.

  • Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.

  • AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.

  • The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.

Notable Quotes

"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."

"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."

"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."

"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."

"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."

"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."

"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."

"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."

"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."

"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."

Ask the Rosenbot
Kevin Bethune
Gatekeepers and Servant Leadership
2020 • DesignOps Community
Daniel Korczynski
Why AI Is Bad at Research (and how to make it actually useful)
2026 • Advancing Research 2026
Gold
Melissa Schmidt
How UX Research Hit It Big in Las Vegas
2019 • Enterprise Experience 2019
Gold
Uday Gajendar
Day 2 Welcome
2024 • Designing with AI 2024
Gold
Bud Caddell
DesignOps Community Sensing Session
2021 • DesignOps Community
Anne Mamaghani
How Your Organization's Generative Workshops Are Probably Going Wrong and How to Get Them Right
2023 • Advancing Research 2023
Gold
Ed Mullen
Designing the Unseen: Enabling Institutions to Build Public Trust
2022 • Civic Design 2022
Gold
Phil Hesketh
Designing Accessible Research Workflows
2021 • DesignOps Summit 2021
Gold
Anna Nguyen
Why Our Voice of the Customer is Better Than Yours
2022 • Advancing Research 2022
Gold
Amy Paris
Delivering Equity: Government Services for All Ages, Languages, Sexual Orientations, and Gender Identities
2021 • Civic Design 2021
Gold
Satyam Kantamneni
Do You Have an Experience Vision?
2023 • Enterprise Community
Marieke McCloskey
User Science: Product Analytics & User Research
2021 • Advancing Research 2021
Gold
Natalie Dunbar
DesignOps and Content Strategy: Envisioning the Future Together
2021 • DesignOps Summit 2021
Gold
Samuel Martin
Co-Design vs Faux-Design: Navigating the Complexities of Sharing Power in Co-Design
2026 • Rosenfeld Community
Sam Proulx
Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
2022 • Advancing Research 2022
Gold
John Devanney
The Design Management Office
2017 • DesignOps Summit 2017
Gold

More Videos

Lavy Kumar

"With how of work now separated from where we work, the design process stays the same—thinking holistically for all personas and touchpoints."

Lavy Kumar Kat Temple Shan Sebastian Tara Jensen Jenn Chou

Future of Work

June 9, 2021

Andreas Huebner

"The pandemic pushed us to be creative, and it’s been fun for users to participate remotely in the design process."

Andreas Huebner Amy Takata Craig Brookes

What Is It Like To Be Part of The UX Team at Compass?

March 11, 2021

Ricardo Martins

"Sixty-one factors influence students’ decisions when choosing a college—that’s a lot more than you might think."

Ricardo Martins

Unlocking the power of advanced quantitative methods

March 12, 2025

Monty Hammontree

"Learning invalidates being right as the currency of research and product decisions."

Monty Hammontree

The Future of UX Research

December 3, 2020

Benjamin Real

"The C-suite often sees big change as resistance to change rather than needs for careful transition and impact understanding."

Benjamin Real

Showing the Value of DesignOps by Not Having a DesignOps Team

October 21, 2020

Kate Towsey

"Every operation is going to have whirlwinds and fires all the time; strategy doesn't end that."

Kate Towsey

Ask Me Anything (AMA) with Kate Towsey

April 2, 2025

Steve Portigal

"Visualizing data in a way executives can interpret is key to shifting from I think to I know."

Steve Portigal Chris Chapo Kelly Goto Christian Rohrer

Discussion

May 13, 2015

Sam Proulx

"If it didn’t come from Amazon or Costco, I probably don’t own it because I know the experience will be consistent."

Sam Proulx

Online Shopping: Designing an Accessible Experience

March 28, 2023

Sam Proulx

"Involving people with disabilities throughout the product development cycle is crucial, from prototype reviews through to acceptance testing."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

October 1, 2021