通过查询似然建模释放 LLM 在稠密检索中的力量
摘要
稠密检索是信息检索（IR）中的关键任务，为下游任务如重排序和增强生成提供了基础。最近，大语言模型（LLMs）在语义理解方面展现了惊人的能力，使其成为关注稠密检索的研究人员的吸引对象。尽管 LLM 作为解码器式生成模型在语言生成方面表现出色，但由于缺乏对后续标记的注意力，往往在建模全局信息方面不足。drawing inspiration from the classical word-based language modeling approach for IR, specifically the query likelihood (QL) model, we aim to leverage the generative strengths of LLMs through QL maximization. Rather than employing QL estimation for document ranking, we propose an auxiliary task of QL maximization to enhance the backbone for subsequent contrastive learning of the retriever. We introduce our model, LLM-QL, which incorporates two key components: Attention Block (AB) and Document Corruption (DC). AB blocks the attention of predictive tokens to the document tokens before the document's ending token, while DC corrupts a document by masking a portion of its tokens during prediction. Evaluations on the in-domain (MS MARCO) and out-of-domain dataset (BEIR) indicate LLM-QL's superiority over other LLM-based retrievers. Furthermore, comprehensive analyses also validate the efficacy of LLM-QL and its components.
引用
@article{arxiv.2504.05216,
title = {Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling},
author = {Hengran Zhang and Keping Bi and Jiafeng Guo and Xiaojie Sun and Shihao Liu and Daiting Shi and Dawei Yin and Xueqi Cheng},
journal= {arXiv preprint arXiv:2504.05216},
year = {2025}
}
备注
12 pages, 3 figures