Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case
Abstract
Background Large language models (LLMs) face challenges in inductive thematic analysis, a task requiring deep interpretive and domain-specific expertise. We evaluated the feasibility of using LLMs to replicate expert-driven thematic analysis of social media data. Methods Using two temporally non-intersecting Reddit datasets on xylazine (n=286 and n=686, for model optimization and validation, respectively) with twelve expert-derived themes, we evaluated five LLMs against expert coding. We modeled the task as a series of binary classifications, rather than a single, multi-label classification, employing zero-, single-, and few-shot prompting strategies and measuring performance via accuracy, precision, recall, and F1-score. Results On the validation set, GPT-4o with two-shot prompting performed best (accuracy: 90.9%; F1-score: 0.71). For high-prevalence themes, model-derived thematic distributions closely mirrored expert classifications (e.g., xylazine use: 13.6% vs. 17.8%; MOUD use: 16.5% vs. 17.8%). Conclusions Our findings suggest that few-shot LLM-based approaches can automate thematic analyses, offering a scalable supplement for qualitative research. Keywords: thematic analysis, large language models, natural language processing, qualitative analysis, social media, prompt engineering, public health
Cite
@article{arxiv.2507.10803,
title = {Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case},
author = {JaMor Hairston and Ritvik Ranjan and Sahithi Lakamana and Anthony Spadaro and Selen Bozkurt and Jeanmarie Perrone and Abeed Sarker},
journal= {arXiv preprint arXiv:2507.10803},
year = {2025}
}
Comments
Pages: 19, Abstract word count: 151 words, Manuscript word count: 2185 words, References: 14, Figures: 3, Tables: 2