English

One Model Is Enough: Native Retrieval Embeddings from LLM Agent Hidden States

Computation and Language 2026-03-10 v1 Artificial Intelligence Information Retrieval

Abstract

LLM agents that retrieve external knowledge typically generate a search query as text, then run a separate embedding model to encode it into a vector. This two-model pipeline adds infrastructure complexity and latency, yet is redundant: the LLM already encodes the full conversational context in its hidden states. We propose equipping LLM agents with native retrieval capability by adding a lightweight projection head that maps hidden states directly into the embedding space, eliminating the need for a separate embedding model. Trained with a combination of alignment, contrastive, and rank distillation losses, our method retains 97\% of baseline retrieval quality while enabling the LLM agent to search with its own representations. Experiments on the QReCC conversational search benchmark show competitive Recall@10 and MRR@10 compared to the standard generate-then-encode pipeline, with systematic ablations confirming the contribution of each loss component.

Keywords

Cite

@article{arxiv.2603.08429,
  title  = {One Model Is Enough: Native Retrieval Embeddings from LLM Agent Hidden States},
  author = {Bo Jiang},
  journal= {arXiv preprint arXiv:2603.08429},
  year   = {2026}
}
R2 v1 2026-07-01T11:10:24.913Z