[2604.00698] Learning to Hint for Reinforcement Learning

Computer Science > Machine Learning arXiv:2604.00698 (cs) [Submitted on 1 Apr 2026] Title:Learning to Hint for Reinforcement Learning Authors:Yu Xia, Canwen Xu, Zhewei Yao, Julian McAuley, Yuxiong He View a PDF of the paper titled Learning to Hint for Reinforcement Learning, by Yu Xia and 4 other authors View PDF HTML (experimental) Abstract:Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards, but it often suffers from advantage collapse: when all rollouts in a group receive the same reward, the group yields zero relative advantage and thus no learning signal. For example, if a question is too hard for the reasoner, all sampled rollouts can be incorrect and receive zero reward. Recent work addresses this issue by adding hints or auxiliary scaffolds to such hard questions so that the reasoner produces mixed outcomes and recovers a non-zero update. However, existing hints are usually fixed rather than adapted to the current reasoner, and a hint that creates learning signal under the hinted input does not necessarily improve the no-hint policy used at test time. To this end, we propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy during RL. For each hard question, the hinter generates hints online conditioned on the current reasoner's incorrect rollout, allowing hint generation to adapt to the reasoner's evolving errors. We further introduce hint r...