English

Can You Detect the Difference?

Computation and Language 2025-07-15 v1 Artificial Intelligence

Abstract

The rapid advancement of large language models (LLMs) has raised concerns about reliably detecting AI-generated text. Stylometric metrics work well on autoregressive (AR) outputs, but their effectiveness on diffusion-based models is unknown. We present the first systematic comparison of diffusion-generated text (LLaDA) and AR-generated text (LLaMA) using 2 000 samples. Perplexity, burstiness, lexical diversity, readability, and BLEU/ROUGE scores show that LLaDA closely mimics human text in perplexity and burstiness, yielding high false-negative rates for AR-oriented detectors. LLaMA shows much lower perplexity but reduced lexical fidelity. Relying on any single metric fails to separate diffusion outputs from human writing. We highlight the need for diffusion-aware detectors and outline directions such as hybrid models, diffusion-specific stylometric signatures, and robust watermarking.

Keywords

Cite

@article{arxiv.2507.10475,
  title  = {Can You Detect the Difference?},
  author = {İsmail Tarım and Aytuğ Onan},
  journal= {arXiv preprint arXiv:2507.10475},
  year   = {2025}
}

Comments

11 pages, 3 figures, 2 tables. Code and data: https://github.com/ismailtrm/ceng_404. Cross-list requested to cs.AI for AI-safety relevance

R2 v1 2026-07-01T04:00:26.472Z