Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Summary
- Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction
- Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks
- Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately.
- Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis
Key Insights
-
•
ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.
-
•
ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.
-
•
Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.
-
•
AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.
-
•
Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.
-
•
Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.
-
•
Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.
-
•
Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.
-
•
AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.
-
•
The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.
Notable Quotes
"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."
"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."
"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."
"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."
"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."
"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."
"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."
"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."
"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."
"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."
Or choose a question:
More Videos
"Our one metric that mattered was maximizing the number of VOС-tagged tickets solved – showing customer feedback made it to the roadmap."
Shipra KayanHow we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
September 30, 2021
"When you’re in a culture that celebrates complexity, making things simple is still a very hard thing to do."
Peter MorvilleThe Architecture of Understanding
May 13, 2015
"Only one internal hire on the UX team was a huge misopportunity for us."
Renee ReidBecoming a ResearchH.E.R (Highly Enterprise Ready)
June 3, 2019
"The computer metaphor of mind does not do a very good job of explaining what we see people do with their bodies."
Dane DeSutterKeeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
March 26, 2024
"A good two-factor system offers multiple methods, like authenticator apps, calls, and texts, with generous timeouts."
Sam ProulxOnline Shopping: Designing an Accessible Experience
October 3, 2023
"Aligning content to audience expectations requires understanding their intent through market research."
Andrew Custage Michael MallettThe Digital Journey: Research on Consumer Frustration and Loyalty
March 29, 2023
"Facilitation is part setting the stage and part performing on it."
Shelby SwitzerMaking Space for Community Knowledge-sharing in a Distributed World
December 10, 2021
"The choices we make about AI are about much more than speed versus quality — they define how we value research and our relationships."
Llewyn PaineCoexisting with AI: A practical guide for researchers to navigate tools, ethics, and integration
March 11, 2025
"Learning is the new knowing, and researchers are catalysts of organizational learning."
Robin BeersNavigating organizational systems: Rethinking researcher’s role in driving change
March 13, 2025
