Long-context language models (LCLMs), characterized by their extensive context window, are becoming popular. However, despite the fact that they are nearly perfect at standard long-context retrieval tasks, our evaluations demonstrate they fail in some basic cases. Later, we find they can be well addressed with a sufficient number of reasoning steps, guided by specific CoT prompts. This result emphasizes the potential necessity of solving specific long-context tasks using long-CoT methods, while previous long-context benchmarks always ignore the necessity of long reasoning for long-context tasks and treat them as direct QA tasks.
@article{arxiv.2410.04422,
title = {Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps},
author = {Yijiong Yu and Yongfeng Huang and Zhixiao Qi and Wei Wang and Weifeng Liu and Ran Chen and Ji Pei},
journal= {arXiv preprint arXiv:2410.04422},
year = {2025}
}
Comments
Our code is publicly available at https://github.com/yuyijiong/hard_retrieval_for_llm and the datasets is at https://huggingface.co/datasets/yuyijiong/difficult_retrieval