[2604.04648] From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism

Computer Science > Machine Learning arXiv:2604.04648 (cs) [Submitted on 6 Apr 2026] Title:From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism Authors:Zhuohao Yu, Zhiwei Steven Wu, Adam Block View a PDF of the paper titled From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism, by Zhuohao Yu and 2 other authors View PDF HTML (experimental) Abstract:Inference-time compute scaling has emerged as a powerful paradigm for improving language model performance on a wide range of tasks, but the question of how best to use the additional compute remains open. A popular approach is BoN sampling, where N candidate responses are generated, scored according to a reward model, and the highest-scoring response is selected. While this approach can improve performance, it is vulnerable to reward hacking, where performance degrades as N increases due to the selection of responses that exploit imperfections in the reward model instead of genuinely improving generation quality. Prior attempts to mitigate reward hacking, via stronger reward models or heavy-handed distributional regularization, either fail to fully address over-optimization or are too conservative to exploit additional compute. In this work, we explore the principle of pessimism in RL, which uses lower confidence bounds on value estimates to avoid OOD actions with uncertain reward estimates. Our approach, termed as caution, can be seen as the reverse of curiosity: where cu...