Offline reinforcement learning (RL) is a variant of RL where the policy is learned from a previously collected dataset of trajectories and rewards. In our work, we propose a practical approach to offline RL with large language models (LLMs). We recast the problem as reward-weighted fine-tuning, which can be solved using similar techniques to supervised fine-tuning (SFT). To showcase the value of our approach, we apply it to learning short-horizon question-answering policies of a fixed length, where the agent reasons about potential answers or asks clarifying questions. Our work stands in a stark contrast to state-of-the-art methods in this domain, based on SFT and direct preference optimization, which have additional hyper-parameters and do not directly optimize for rewards. We compare to them empirically, and report major gains in both optimized rewards and language quality.
@article{arxiv.2506.06964,
title = {Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization},
author = {Subhojyoti Mukherjee and Viet Dac Lai and Raghavendra Addanki and Ryan Rossi and Seunghyun Yoon and Trung Bui and Anup Rao and Jayakumar Subramanian and Branislav Kveton},
journal= {arXiv preprint arXiv:2506.06964},
year = {2026}
}
Comments
Advances in Neural Information Processing Systems 38