[2604.00200] Offline Constrained RLHF with Multiple Preference Oracles

Computer Science > Machine Learning arXiv:2604.00200 (cs) [Submitted on 31 Mar 2026] Title:Offline Constrained RLHF with Multiple Preference Oracles Authors:Brenden Latham, Mehrdad Moharrami View a PDF of the paper titled Offline Constrained RLHF with Multiple Preference Oracles, by Brenden Latham and Mehrdad Moharrami View PDF HTML (experimental) Abstract:We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization. Subjects: Machine Learning (cs.LG) Cite as: arXiv:2604.00200 [cs.LG]   (or arXiv:2604.00200v1 [cs.LG] for this version)   https://doi.org/10.48550/arXiv.2604.00200 Focus to learn more arXiv-i...