English

HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models

Computer Vision and Pattern Recognition 2026-04-15 v1

Abstract

Large vision-language models (LVLMs) achieve strong multimodal performance, but still suffer from hallucinations caused by unstable visual grounding and over-reliance on language priors. Existing training-free decoding methods typically apply calibration at every decoding step, introducing unnecessary computation and potentially disrupting stable predictions. We address this problem by identifying layer-wise hesitation, a simple signal of grounding instability reflected by fluctuations in token preference across intermediate layers. Based on this observation, we propose Hesitation-Triggered Differential Calibration (HTDC), a training-free decoding framework that preserves standard full-branch inference and activates calibration only at hesitation-prone steps. When triggered, HTDC contrasts the full branch with two lightweight probes, a visual-nullification probe and a semantic-nullification probe, to suppress hallucination-prone candidates while avoiding unnecessary intervention on stable steps. Experiments on representative hallucination benchmarks show that HTDC consistently reduces hallucinations while maintaining strong task accuracy, achieving a favorable trade-off between effectiveness and computational overhead.

Keywords

Cite

@article{arxiv.2604.12115,
  title  = {HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models},
  author = {Xinyun Liu},
  journal= {arXiv preprint arXiv:2604.12115},
  year   = {2026}
}

Comments

10 pages, 4 figures, 6 tables

R2 v1 2026-07-01T12:07:41.544Z