RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning

Aleksei Arzhantsev; Otmane Sakhi; Flavian Vasile

RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning

Machine Learning 2025-10-06 v1

Authors: Aleksei Arzhantsev , Otmane Sakhi , Flavian Vasile

Abstract

Reinforcement learning (RL) is central to improving reasoning in large language models (LLMs) but typically requires ground-truth rewards. Test-Time Reinforcement Learning (TTRL) removes this need by using majority-vote rewards, but relies on heavy online RL and incurs substantial computational cost. We propose RoiRL: Reasoning with offline iterative Reinforcement Learning, a family of lightweight offline learning alternatives that can target the same regularized optimal policies. Unlike TTRL, RoiRL eliminates the need to maintain a reference model and instead optimizes weighted log-likelihood objectives, enabling stable training with significantly lower memory and compute requirements. Experimental results show that RoiRL trains to 2.5x faster and consistently outperforms TTRL on reasoning benchmarks, establishing a scalable path to self-improving LLMs without labels.

Keywords

reinforcement learning reinforcement fine-tuning reinforcement learning from human feedback

Cite

@article{arxiv.2510.02892,
  title  = {RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning},
  author = {Aleksei Arzhantsev and Otmane Sakhi and Flavian Vasile},
  journal= {arXiv preprint arXiv:2510.02892},
  year   = {2025}
}

Comments

Accepted to the Efficient Reasoning Workshop at NeuRIPS 2025

RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning

Abstract

Keywords

Cite

Comments

Related papers