[2511.04439] CoRPO: Adding a Correctness Bias to GRPO Improves Generalization

Computer Science > Artificial Intelligence arXiv:2511.04439 (cs) [Submitted on 6 Nov 2025 (v1), last revised 4 Mar 2026 (this version, v3)] Title:CoRPO: Adding a Correctness Bias to GRPO Improves Generalization Authors:Anisha Garg, Claire Zhang, Nishit Neema, David Bick, Ganesh Venkatesh, Joel Hestness View a PDF of the paper titled CoRPO: Adding a Correctness Bias to GRPO Improves Generalization, by Anisha Garg and 5 other authors View PDF HTML (experimental) Abstract:Group-Relative Policy Optimization (GRPO) has emerged as the standard for training reasoning capabilities in large language models through reinforcement learning. By estimating advantages using group-mean rewards rather than a learned critic, GRPO has enabled efficient scaling of reinforcement learning from verifiable rewards (RLVR). However, we identify a fundamental limitation: GRPO's mean baseline can assign positive advantages to incorrect solutions simply because they outperform a poorly-performing group average. It leads to overestimation of advantages and reinforcement of incorrect behaviours. To address this, we propose Correctness-Relative Policy Optimization (CoRPO), a simple modification to the GRPO objective that clips the minimum baseline to a fixed correctness threshold. We show that baseline clipping introduces a protective bias to advantage estimation that mitigates overfitting while preserving effective exploration. Empirically, CoRPO-trained models improve cross-domain reasoning, generalizi...