Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences. Methods such as RiC that directly inject raw reward values into prompts face significant numerical sensitivity issues--for instance, LLMs may fail to distinguish between 9.11 and 9.8--while alternatives like MORLHF, Rewarded Soups, and MODPO incur high computational costs by training multiple models. In this work, we introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations. Our approach leverages a diverse set of strictly increasing, non-linear utility functions to transform user-specified preferences into symbolic tokens, which are then used to condition a single LLM. This design not only mitigates numerical reasoning challenges but also substantially reduces training overhead, yielding models that achieve superior Pareto fronts and robust alignment across complex reward dimensions.
@article{arxiv.2503.10669,
title = {UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality},
author = {Zelei Cheng and Xin-Qiang Cai and Yuting Tang and Pushi Zhang and Boming Yang and Masashi Sugiyama and Xinyu Xing},
journal= {arXiv preprint arXiv:2503.10669},
year = {2025}
}
Comments
Language Modeling, Machine Learning for NLP, Distributional Pareto-Optimal