ContraSolver：通过解决内部偏好矛盾实现语言模型的自我对齐
摘要
尽管在开发大型语言模型（LLM）方面取得了显著进展，但对其行为的控制仍然具有挑战性。直接偏好优化（DPO）假设存在一个隐含的奖励函数来评估LLM的响应。这种假设表明，对相同输入的不同响应存在严格的偏好排序。然而，根据我们的实验观察，LLM中始终存在偏好矛盾。本文构建了一个表示不同响应之间偏好关系的图结构，通过自我标注来寻找偏好顺序中的矛盾。我们提出了ContraSolver—an algorithm that traverses all edges on the preference graph to identify those that might cause contradictions. ContraSolver initializes the graph with a maximum spanning tree and identifies contradictory edges, prioritizing the resolution of low-confidence preferences while preserving high-confidence ones. Experimental results on four different generation tasks show that the performance of different LLMs can be largely improved through our completely unsupervised self-alignment. Furthermore, by analyzing the preference graphs of LLMs with and without self-alignment by ContraSolver, we quantify the reduction in contradictions, suggesting that resolving preference contradictions is crucial for achieving better alignment performance.
引用
@article{arxiv.2406.08842,
title = {ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions},
author = {Xu Zhang and Xunjian Yin and Xiaojun Wan},
journal= {arXiv preprint arXiv:2406.08842},
year = {2024}
}