Large language models (LLMs) are increasingly being deployed in cost and latency-sensitive settings. While chain-of-thought improves reasoning, it can waste tokens on simple requests. We study selective thinking for tool-using LLMs and introduce Adaptive Rejection Sampling (Ada-RS), an algorithm-agnostic sample filtering framework for learning selective and efficient reasoning. For each given context, Ada-RS scores multiple sampled completions with an adaptive length-penalized reward then applies stochastic rejection sampling to retain only high-reward candidates (or preference pairs) for downstream optimization. We demonstrate how Ada-RS plugs into both preference pair (e.g. DPO) or grouped policy optimization strategies (e.g. DAPO). Using Qwen3-8B with LoRA on a synthetic tool call-oriented e-commerce benchmark, Ada-RS improves the accuracy-efficiency frontier over standard algorithms by reducing average output tokens by up to 80% and reducing thinking rate by up to 95% while maintaining or improving tool call accuracy. These results highlight that training-signal selection is a powerful lever for efficient reasoning in latency-sensitive deployments.
@article{arxiv.2602.19519,
title = {Ada-RS: Adaptive Rejection Sampling for Selective Thinking},
author = {Yirou Ge and Yixi Li and Alec Chiu and Shivani Shekhar and Zijie Pan and Avinash Thangali and Yun-Shiuan Chuang and Chaitanya Kulkarni and Uma Kona and Linsey Pang and Prakhar Mehrotra},
journal= {arXiv preprint arXiv:2602.19519},
year = {2026}
}