Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Summary
Following the emergence of Generative AI as a potential revolution in the UX field, a great deal of AI-driven tools arose to enhance the efficiency of UX research, including data analysis. Qualitative data analysis is a process that conventionally relies on human intelligence to discern patterns, establish connections, and derive actionable insights and frameworks. Many studies have involved comparing the quality of qualitative analyses generated by humans with those produced by AI language models like ChatGPT (Hamilton et al., 2023). Despite the undeniable appeal of automation and speed, there is ongoing debate about AI’s ability to replace human intelligence in qualitative analysis, which may be unlikely at this moment. Then the question is: To what extent can AI contribute to qualitative data analysis? In this case study, I delved into the thematic analysis and post-analysis stage, i.e. synthesizing insights into a framework. Framework, in this context, refers to a conceptual structure that illustrates the components of a human experience and how the components interconnect and operate within the structure. It is a concise model that encapsulates the entirety of research insights. The topic of my case study is "trust relationships between job seekers and hirers in the marketplace,, aligning with the business focus of my company. From my secondary research, I found that, ChatGPT needed multiple rounds of training using diverse prompts to conduct precise and comprehensive thematic analysis. ChatGPT can execute fine-quality thematic analysis under the help of right prompts, yet it falls short in replacing human intelligence for synthesizing insights and crafting frameworks for engaging narratives. Its limitation lies in lacking the depth of contextual understanding within a company, such as understanding what’s missing from the company’s mainstream discourse to create a human-centered story based on data analysis. To craft a framework that conveys good storytelling and organizational impact, it requires the researcher's introspection into knowledge gaps in the specific organizational context. Thus, the best practice is to combine human interpretation and AI production. In my talk, I will demonstrate several principles to guide this practice. Takeaways We’ll cover principles of how to employ ChatGPT in qualitative analysis, specifically focusing on its application in synthesizing and crafting frameworks that convey compelling and insightful narratives: Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately. Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis
Key Insights
-
•
ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.
-
•
ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.
-
•
Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.
-
•
AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.
-
•
Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.
-
•
Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.
-
•
Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.
-
•
Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.
-
•
AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.
-
•
The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.
Notable Quotes
"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."
"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."
"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."
"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."
"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."
"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."
"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."
"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."
"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."
"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."
Or choose a question:
More Videos
"With how of work now separated from where we work, the design process stays the same—thinking holistically for all personas and touchpoints."
Lavy Kumar Kat Temple Shan Sebastian Tara Jensen Jenn ChouFuture of Work
June 9, 2021
"The pandemic pushed us to be creative, and it’s been fun for users to participate remotely in the design process."
Andreas Huebner Amy Takata Craig BrookesWhat Is It Like To Be Part of The UX Team at Compass?
March 11, 2021
"Sixty-one factors influence students’ decisions when choosing a college—that’s a lot more than you might think."
Ricardo MartinsUnlocking the power of advanced quantitative methods
March 12, 2025
"Learning invalidates being right as the currency of research and product decisions."
Monty HammontreeThe Future of UX Research
December 3, 2020
"The C-suite often sees big change as resistance to change rather than needs for careful transition and impact understanding."
Benjamin RealShowing the Value of DesignOps by Not Having a DesignOps Team
October 21, 2020
"Every operation is going to have whirlwinds and fires all the time; strategy doesn't end that."
Kate TowseyAsk Me Anything (AMA) with Kate Towsey
April 2, 2025
"Visualizing data in a way executives can interpret is key to shifting from I think to I know."
Steve Portigal Chris Chapo Kelly Goto Christian RohrerDiscussion
May 13, 2015
"If it didn’t come from Amazon or Costco, I probably don’t own it because I know the experience will be consistent."
Sam ProulxOnline Shopping: Designing an Accessible Experience
March 28, 2023
"Involving people with disabilities throughout the product development cycle is crucial, from prototype reviews through to acceptance testing."
Sam ProulxPrototype Reviews, People With Disabilities, and You
October 1, 2021
