Large Language Models (LLMs) are increasingly used as automated annotators to scale dataset creation, yet their reliability as unbiased annotators--especially for low-resource and identity-sensitive settings--remains poorly understood. In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences. We conduct a systematic benchmark of 17 LLMs using a unified evaluation framework. Our analysis uncovers annotator bias and substantial instability in model judgments. Surprisingly, increased model scale does not guarantee improved annotation quality--smaller, more task-aligned models frequently exhibit more consistent behavior than their larger counterparts. These results highlight important limitations of current LLMs for sensitive annotation tasks in low-resource languages and underscore the need for careful evaluation before deployment.
@article{arxiv.2602.16241,
title = {Are LLMs Ready to Replace Bangla Annotators?},
author = {Md. Najib Hasan and Touseef Hasan and Souvika Sarkar},
journal= {arXiv preprint arXiv:2602.16241},
year = {2026}
}
Comments
We have identified significant methodological discrepancies in the current version of the manuscript that affect the validity and reproducibility of the reported results. In order to prevent potential misunderstanding or misinterpretation of our findings, we request the complete withdrawal of this submission while we conduct a thorough revision