Large language models (LLMs) have recently demonstrated remarkable success in mathematical reasoning. Despite progress in methods like chain-of-thought prompting and self-consistency sampling, these advances often focus on final correctness without ensuring that the underlying reasoning process is coherent and reliable. This paper introduces Step-KTO, a training framework that combines process-level and outcome-level binary feedback to guide LLMs toward more trustworthy reasoning trajectories. By providing binary evaluations for both the intermediate reasoning steps and the final answer, Step-KTO encourages the model to adhere to logical progressions rather than relying on superficial shortcuts. Our experiments on challenging mathematical benchmarks show that Step-KTO significantly improves both final answer accuracy and the quality of intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO achieves a notable improvement in Pass@1 accuracy over strong baselines. These results highlight the promise of integrating stepwise process feedback into LLM training, paving the way toward more interpretable and dependable reasoning capabilities.
@article{arxiv.2501.10799,
title = {Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback},
author = {Yen-Ting Lin and Di Jin and Tengyu Xu and Tianhao Wu and Sainbayar Sukhbaatar and Chen Zhu and Yun He and Yun-Nung Chen and Jason Weston and Yuandong Tian and Arash Rahnama and Sinong Wang and Hao Ma and Han Fang},
journal= {arXiv preprint arXiv:2501.10799},
year = {2025}
}