Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an explicit link with a specific normative model of human choice. Our paper elevates this connection to the full generality of DPO's normative framework. Getting there requires reworking human choice theory's textbook path for a better RLHF/ML fit. It elevates the connection to a remarkably broad viewpoint on preference optimization, considering the current panorama of DPO follow-ups. It also unveils unexpected riches for ML, chief among which the support for non-convex losses, the fact that any compliant ML analytical choice can be embedded with any human choice model, and a normative framework's umbrella wide enough to safeguard DPO's extensions (margins, length correction, ...). A toy experiment ``far away'' from the DPO crowd is given.
@article{arxiv.2507.07855,
title = {DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory},
author = {Wenxuan Zhou and Shujian Zhang and Brice Magdalou and John Lambert and Ehsan Amid and Richard Nock and Andrew Hard},
journal= {arXiv preprint arXiv:2507.07855},
year = {2026}
}