[2503.15477] What Makes a Reward Model a Good Teacher? An Optimization Perspective

Computer Science > Machine Learning arXiv:2503.15477 (cs) [Submitted on 19 Mar 2025 (v1), last revised 27 Feb 2026 (this version, v4)] Title:What Makes a Reward Model a Good Teacher? An Optimization Perspective Authors:Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, Sanjeev Arora View a PDF of the paper titled What Makes a Reward Model a Good Teacher? An Optimization Perspective, by Noam Razin and 5 other authors View PDF HTML (experimental) Abstract:The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primarily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes a reward model an effective teacher. We address this question from an optimization perspective. First, we prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape. Consequently, even a perfectly accurate reward model can lead to extremely slow optimization, underperforming less accurate models that induce higher reward variance. We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another. These results establish a fundamental limitation of evaluating reward models solely based on accuracy or independently of the language model they guide. Experiments using models of up to 8B...