![[2511.00794] Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2511.00794] Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
Summary
This paper presents PREPO, a novel approach to enhance data efficiency in reinforcement learning for large language models by leveraging intrinsic data properties.
Why It Matters
As reinforcement learning continues to evolve, improving data efficiency is crucial for optimizing training processes, especially for large language models. This study offers insights into how intrinsic exploration can reduce computational costs while maintaining performance, making it relevant for researchers and practitioners in AI and machine learning.
Key Takeaways
- PREPO improves data efficiency in reinforcement learning for large language models.
- The method uses prompt perplexity to guide model learning from simple to complex contexts.
- Differentiating rollout sequences by their relative entropy enhances exploration.
- The approach achieves up to three times fewer rollouts while maintaining performance.
- Theoretical analyses support the effectiveness of the proposed method.
Computer Science > Machine Learning arXiv:2511.00794 (cs) [Submitted on 2 Nov 2025 (v1), last revised 19 Feb 2026 (this version, v3)] Title:Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration Authors:Yan Sun, Jia Guo, Stanley Kok, Zihao Wang, Zujie Wen, Zhiqiang Zhang View a PDF of the paper titled Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration, by Yan Sun and 5 other authors View PDF HTML (experimental) Abstract:Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models, yet training remains costly because many rollouts contribute little to optimization, considering the amount of computation required. This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR. We propose PREPO with two complementary components. First, we adopt prompt perplexity as an indicator of model adaptability in learning, enabling the model to progress from well-understood contexts to more challenging ones. Second, we amplify the discrepancy among the rollouts by differentiating their relative entropy, and prioritize sequences that exhibit a higher degree of exploration. Together, these mechanisms reduce rollout demand while preserving competitive performance. On the Qwen and Llama models, PREPO achieves effective results on mathematical reasoning benchmarks with up to 3 times fewer roll...
