[2510.00819] Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning

Computer Science > Machine Learning arXiv:2510.00819 (cs) [Submitted on 1 Oct 2025 (v1), last revised 28 Feb 2026 (this version, v2)] Title:Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning Authors:Luckeciano C. Melo, Alessandro Abate, Yarin Gal View a PDF of the paper titled Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning, by Luckeciano C. Melo and 2 other authors View PDF HTML (experimental) Abstract:Reinforcement Learning, particularly through policy gradient methods, has played a central role in enabling reasoning capabilities of Large Language Models. However, the optimization stability of policy gradients in this setting remains understudied. As a result, existing implementations often resort to conservative hyperparameter choices to ensure stability, which requires more training samples and increases computational costs. Hence, developing models for reliably tracking the underlying optimization dynamics and leveraging them into training enables more sample-efficient regimes and further unleashes scalable post-training. We address this gap by formalizing the stochastic optimization problem of policy gradients with explicit consideration of second-order geometry. We propose a tractable computational framework that tracks and leverages curvature information during policy updates. We further employ this framework to design interventions in the optimization process through data selection. T...