![[2602.23116] Regularized Online RLHF with Generalized Bilinear Preferences](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.23116] Regularized Online RLHF with Generalized Bilinear Preferences
Summary
This paper explores contextual online Reinforcement Learning with Human Feedback (RLHF) using a Generalized Bilinear Preference Model to achieve efficient learning in high-dimensional spaces.
Why It Matters
The research addresses the challenges of preference learning in online RLHF, providing new algorithms that improve regret bounds. This is crucial for advancing AI systems that rely on human feedback, particularly in complex environments where traditional methods may struggle.
Key Takeaways
- Introduces a Generalized Bilinear Preference Model for contextual online RLHF.
- Proves that the dual gap of the greedy policy is bounded by estimation error.
- Presents two algorithms with improved regret bounds for online RLHF.
- Demonstrates statistical efficiency in high-dimensional settings.
- Generalizes beyond previous works limited to reverse KL-regularization.
Computer Science > Machine Learning arXiv:2602.23116 (cs) [Submitted on 26 Feb 2026] Title:Regularized Online RLHF with Generalized Bilinear Preferences Authors:Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun View a PDF of the paper titled Regularized Online RLHF with Generalized Bilinear Preferences, by Junghyun Lee and 4 other authors View PDF HTML (experimental) Abstract:We consider the problem of contextual online RLHF with general preferences, where the goal is to identify the Nash Equilibrium. We adopt the Generalized Bilinear Preference Model (GBPM) to capture potentially intransitive preferences via low-rank, skew-symmetric matrices. We investigate general preference learning with any strongly convex regularizer (where $\eta^{-1}$ is the regularization strength), generalizing beyond prior works limited to reverse KL-regularization. Central to our analysis is proving that the dual gap of the greedy policy is bounded by the square of the estimation error - a result derived solely from strong convexity and the skew-symmetricity of this http URL on this insight and a feature diversity assumption, we establish two regret bounds via two simple algorithms: (1) Greedy Sampling achieves polylogarithmic, $e^{O(\eta)}$-free regret $\tilde{O}(\eta d^4 (\log T)^2)$. (2) Explore-Then-Commit achieves $\mathrm{poly}(d)$-free regret $\tilde{O}(\sqrt{\eta r T})$ by exploiting the low-rank structure; this is the first statistically efficient guarantee for online R...