[2510.02245] ExGRPO: Learning to Reason from Experience

Computer Science > Machine Learning arXiv:2510.02245 (cs) [Submitted on 2 Oct 2025 (v1), last revised 28 Feb 2026 (this version, v2)] Title:ExGRPO: Learning to Reason from Experience Authors:Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F. Wong, Yu Cheng View a PDF of the paper titled ExGRPO: Learning to Reason from Experience, by Runzhe Zhan and 7 other authors View PDF HTML (experimental) Abstract:Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, wit...