Recent advancements in long chain-of-thoughts(long CoTs) have significantly improved the reasoning capabilities of large language models(LLMs). Existing work finds that the capability of long CoT reasoning can be efficiently elicited by tuning on only a few examples and can easily transfer to other tasks. This motivates us to investigate whether long CoT reasoning is a general capability for LLMs. In this work, we conduct an empirical analysis for this question from the perspective of representation. We find that LLMs do encode long CoT reasoning as a general capability, with a clear distinction from vanilla CoTs. Furthermore, domain-specific representations are also required for the effective transfer of long CoT reasoning. Inspired by these findings, we propose GLoRE, a novel representation engineering method to unleash the general long CoT reasoning capabilities of LLMs. Extensive experiments demonstrate the effectiveness and efficiency of GLoRE in both in-domain and cross-domain scenarios.
@article{arxiv.2503.11314,
title = {Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering},
author = {Xinyu Tang and Xiaolei Wang and Zhihao Lv and Yingqian Min and Wayne Xin Zhao and Binbin Hu and Ziqi Liu and Zhiqiang Zhang},
journal= {arXiv preprint arXiv:2503.11314},
year = {2025}
}