RL alignment methods, including RLHF and DPO, are primarily based on pairwise preference data. Although scalar or score-based feedback has been collected in some settings, it is rarely used directly, and preference magnitude information is typically ignored. Furthermore, current alignment frameworks offer limited capability for unifying heterogeneous supervision signals, making it difficult to jointly leverage diverse data types within a single training paradigm. This limitation constrains the richness and scalability of the alignment process. To address this gap, we propose a \textbf{UN}ified \textbf{A}lignment (UNA) framework capable of training across different types of feedback, including binary, pairwise, and score-based, through a generalized implicit reward function. The reward function is theoretically proved to be the optimal policy by the log sum inequality. Extensive experiments on classical benchmarks consistently demonstrate the advantage of the proposed unified framework with typical LLM base models.
@article{arxiv.2408.15339,
title = {UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Types},
author = {Zhichao Wang and Bin Bi and Can Huang and Shiva Kumar Pentyala and Zixu James Zhu and Sitaram Asur and Na Claire Cheng and Cheng Wan and Dong Nie and Lingzi Hong},
journal= {arXiv preprint arXiv:2408.15339},
year = {2026}
}