[2604.02341] LLM Reasoning with Process Rewards for Outcome-Guided Steps

Computer Science > Machine Learning arXiv:2604.02341 (cs) [Submitted on 8 Feb 2026] Title:LLM Reasoning with Process Rewards for Outcome-Guided Steps Authors:Mohammad Rezaei, Jens Lehmann, Sahar Vahdati View a PDF of the paper titled LLM Reasoning with Process Rewards for Outcome-Guided Steps, by Mohammad Rezaei and 2 other authors View PDF HTML (experimental) Abstract:Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final answers can be checked automatically and converted into reliable training signals. Most such pipelines optimize outcome correctness only, which yields sparse feedback for long, multi-step solutions and offers limited guidance on intermediate reasoning errors. Recent work therefore introduces process reward models (PRMs) to score intermediate steps and provide denser supervision. In practice, PRM scores are often imperfectly aligned with final correctness and can reward locally fluent reasoning that still ends in an incorrect answer. When optimized as absolute rewards, such signals can amplify fluent failure modes and induce reward hacking. We propose PROGRS, a framework that leverages PRMs while keeping outcome correctness dominant. PROGRS treats process rewards as relative preferences within outcome groups rather than absolute targets. We introduce outcome-conditioned centering, which shifts PRM scores of incorrect trajectories to have zero mean within each prompt grou...