MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
Abstract
Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems. However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, cross-encoder ranking systems often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information. To this end, we propose MixLM, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of cross-encoder rankers. In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use mix-interaction, a mixture of text and embedding tokens to represent the input. Specifically, MixLM encodes all items in the catalog into a few embedding tokens and stores in a nearline cache. The encoded item descriptions are used during online inference, effectively reducing the item length from a few thousand text tokens to a few embedding tokens. We share insights from deploying our MixLM framework to a real-world search application at LinkedIn, including a detailed discussion of our training pipelines, as well as a thorough analysis of our online serving infrastructure optimization. With the same latency budget and on-par relevance metrics, MixLM increased throughput by 10.0x comparing with strong baselines, 75.9x over full-text LLM rerankers. The efficiency gains delivered by MixLM enabled full-traffic deployment of LLM-powered search, which resulted in a significant 0.47\% increase in Daily Active Users (DAU) in online A/B tests.
Cite
@article{arxiv.2512.07846,
title = {MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction},
author = {Guoyao Li and Ran He and Shusen Jing and Kayhan Behdin and Yubo Wang and Sundara Raman Ramachandran and Chanh Nguyen and Jian Sheng and Xiaojing Ma and Chuanrui Zhu and Sriram Vasudevan and Muchen Wu and Sayan Ghosh and Lin Su and Qingquan Song and Xiaoqing Wang and Zhipeng Wang and Qing Lan and Yanning Chen and Jingwei Wu and Luke Simon and Wenjing Zhang and Qi Guo and Fedor Borisyuk},
journal= {arXiv preprint arXiv:2512.07846},
year = {2026}
}