Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

Fahim Faisal; Md Mushfiqur Rahman; Antonios Anastasopoulos

Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

Computation and Language 2024-11-19 v1

Authors: Fahim Faisal , Md Mushfiqur Rahman , Antonios Anastasopoulos

Abstract

There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators ("LLM-as-a-judge") is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive in handling both multilingual and dialectal variations. However, if we have to rank the consistency, the weakest area is LLM-human agreement, followed by dialectal consistency. Code repository: \url{https://github.com/ffaisal93/dialect_toxicity_llm_judge}

Keywords

toxicity detection dialectal variations LLM-as-a-judge multilingual NLP consistency evaluation

Cite

@article{arxiv.2411.10954,
  title  = {Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties},
  author = {Fahim Faisal and Md Mushfiqur Rahman and Antonios Anastasopoulos},
  journal= {arXiv preprint arXiv:2411.10954},
  year   = {2024}
}

Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

Abstract

Keywords

Cite

Related papers