English

DiffImaginE: Imagine to Verify Entity Types with Diffusio

Artificial Intelligence 2026-08-04 v1

Abstract

Multimodal named entity recognition (MNER) determines whether each candidate span and entity-type hypothesis is supported by joint textual and visual evidence. Existing imagine-and-compare verifiers map each (span, type) pair to one predicted visual feature, compressing diverse visual realisations into a single prototype and providing a compatibility score without explicit probabilistic semantics. We introduce DiffImaginE, which formulates MNER type verification as conditional latent diffusion inference. Given span-localised visual evidence, a type-conditioned denoiser predicts noise injected into its standardised latent. The resulting denoising error provides an ELBO-consistent surrogate for type-conditional negative log-likelihood, allowing competing type hypotheses to be ranked by how well they explain the observation. DiffImaginE retains a standard multimodal encoder stack and replaces the deterministic verifier with a classifier-free-guided diffusion scorer trained using Min-SNR weighting. We directly supervise per-type diffusion scores as classification logits, learn aggregation across noise levels, and use antithetic sampling to reduce Monte Carlo comparison variance. Our analysis shows that classifier-free guidance sharpens the induced type posterior and characterises when antithetic pairing reduces variance at equal denoiser cost. Experiments on Twitter-2015 and Twitter-2017 show consistent gains over a matched deterministic ImaginE control under the same encoder, auxiliary objectives, and evaluation protocol, supported by ablations and paired significance tests.

Cite

@article{arxiv.2608.03025,
  title  = {DiffImaginE: Imagine to Verify Entity Types with Diffusio},
  author = {Feng Zhang and Feiyu Han and Rongxin Yang and Yang Liu and Yancheng Chen and Rui Wang and Yingguang Yang and Tian Xueyun and Chongyang Zhang and Hao Zheng and Xu Kefu and Congjing Ran and Fuhai Chen and Bin Chong},
  journal= {arXiv preprint arXiv:2608.03025},
  year   = {2026}
}