![[2510.03269] General Exploratory Bonus for Optimistic Exploration in RLHF](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2510.03269] General Exploratory Bonus for Optimistic Exploration in RLHF
Summary
This paper introduces the General Exploratory Bonus (GEB) framework, which enhances optimistic exploration in reinforcement learning with human feedback, addressing biases in current methods.
Why It Matters
Optimistic exploration is crucial for improving sample efficiency in reinforcement learning, particularly when human feedback is involved. The GEB framework offers a theoretically sound and practical approach to overcoming existing biases, potentially leading to more effective AI systems.
Key Takeaways
- Current exploratory bonus methods often reinforce conservative behavior.
- GEB provides a principled solution that satisfies the optimism principle.
- Empirical results show GEB outperforms existing methods on alignment tasks.
- The framework extends across the full alpha-divergence family.
- GEB unifies prior heuristic bonuses, enhancing its applicability.
Computer Science > Machine Learning arXiv:2510.03269 (cs) [Submitted on 27 Sep 2025 (v1), last revised 17 Feb 2026 (this version, v4)] Title:General Exploratory Bonus for Optimistic Exploration in RLHF Authors:Wendi Li, Changdae Oh, Sharon Li View a PDF of the paper titled General Exploratory Bonus for Optimistic Exploration in RLHF, by Wendi Li and 2 other authors View PDF HTML (experimental) Abstract:Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide a theoretical analysis showing that current formulations, under KL or $\alpha$-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discovery of uncertain regions. To address this pitfall, we introduce the General Exploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference-dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full $\alpha$-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both a principled and practical solu...
