Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Summary
- Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction
- Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks
- Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately.
- Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis
Key Insights
-
•
ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.
-
•
ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.
-
•
Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.
-
•
AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.
-
•
Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.
-
•
Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.
-
•
Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.
-
•
Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.
-
•
AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.
-
•
The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.
Notable Quotes
"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."
"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."
"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."
"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."
"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."
"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."
"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."
"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."
"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."
"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."
Or choose a question:
More Videos
"Product managers are going to have to say no to roughly 95% of everything that comes in."
Rich MironovHow Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
December 6, 2022
"Every talk you hear today couldn’t get more personal to our community."
Bria AlexanderTheme Two Intro
October 3, 2023
"Frameworks are a team sport; each player brings unique strengths that help create innovative solutions."
Ruzanna RozmanGetting in Flow with Your Team
January 8, 2024
"An ops-focused discussion guide helps us gather feedback from leadership and designers so our recommendations meet their needs."
Briana Thomas Christina RodriguezWhen Design Ops Comes in H.O.T. : A Tale of a Transformed Design Org
September 30, 2021
"This new vets.gov site is extremely easy. I have not had this level of ease with a government website at all."
Marina MartinLives on the Line: The Stakes of UX at the Scale of Government
June 14, 2018
"Scaling up research impact means turning visibility way up and opening access despite some risk of misuse."
Jake BurghardtStop wasting research: Create new value with insight summaries
July 9, 2025
"Ops-first means standardizing processes so we can scale and onboard new team members quickly."
Clemens Janssen Jane DavisEfficiently Scaling Research as a Team of One
March 28, 2023
"Organizational information models need to grow and be managed, not treated as static deliverables."
Bram WesselEnterprise Information Architecture
January 10, 2020
"Highly trained professionals still default to a presumed 'standard' under pressure, revealing how bias persists."
Kaitlin TaskerFast and Fearless Inclusive Research
March 27, 2023
