Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the \emph{answers} being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling \emph{answer-side} uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.
@article{arxiv.2601.17329,
title = {Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment},
author = {Tiejin Chen and Xiaoou Liu and Vishnu Nandam and Kuan-Ru Liou and Hua Wei},
journal= {arXiv preprint arXiv:2601.17329},
year = {2026}
}