Reasoning-capable large language models (LLMs) achieve strong performance on complex tasks but often exhibit overthinking after distillation, generating unnecessarily long chain-of-thought (CoT) reasoning even for simple inputs and incurring high inference cost. However, naively shortening reasoning length can degrade reasoning accuracy, as concise reasoning may be insufficient for certain inputs and lacks explicit supervision. We propose Auto Long-Short Reasoning (AutoL2S), a distillation framework that empowers non-reasoning LLMs to think thoroughly but only when necessary. AutoL2S first learns a lightweight switching token with verified long-short CoTs to enable instance-wise long-short reasoning selection. Then it leverages long-short reasoning rollouts induced by a switching token in a GRPO-style loss to improve reasoning efficiency while maintaining accuracy. Experiments demonstrate that AutoL2S effectively reduces reasoning length up to 71% with minimal accuracy loss, yielding markedly better trade-off in token length and inference time while preserving accuracy.
@article{arxiv.2505.22662,
title = {AutoL2S: Auto Long-Short Reasoning for Efficient Large Language Models},
author = {Feng Luo and Yu-Neng Chuang and Guanchu Wang and Hoang Anh Duy Le and Shaochen Zhong and Hongyi Liu and Jiayi Yuan and Yang Sui and Vladimir Braverman and Vipin Chaudhary and Xia Hu},
journal= {arXiv preprint arXiv:2505.22662},
year = {2026}
}