This paper investigates bias in GLLM annotations by conceptually replicating manual annotations of Boukes (2024). Using various GLLMs (Llama3.1:8b, Llama3.3:70b, GPT4o, Qwen2.5:72b) in combination with five different prompts for five concepts (political content, interactivity, rationality, incivility, and ideology). We find GLLMs perform adequate in terms of F1 scores, but differ from manual annotations in terms of prevalence, yield substantively different downstream results, and display systematic bias in that they overlap more with each other than with manual annotations. Differences in F1 scores fail to account for the degree of bias.
@article{arxiv.2512.08404,
title = {Are generative AI text annotations systematically biased?},
author = {Sjoerd B. Stolwijk and Mark Boukes and Damian Trilling},
journal= {arXiv preprint arXiv:2512.08404},
year = {2025}
}
Comments
9 pages, 6 figures, 1 table; version submitted to the International Communication Association Annual Conference in Cape Town 2026