[2506.05619] Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

Computer Science > Artificial Intelligence arXiv:2506.05619 (cs) [Submitted on 5 Jun 2025 (v1), last revised 2 Mar 2026 (this version, v3)] Title:Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework Authors:Kihyun Kim, Jiawei Zhang, Asuman Ozdaglar, Pablo A. Parrilo View a PDF of the paper titled Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework, by Kihyun Kim and 3 other authors View PDF HTML (experimental) Abstract:Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment ...