![[2602.20722] Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20722] Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Summary
This paper introduces Batch Adaptation Policy Optimization (BAPO), an off-policy reinforcement learning framework designed to enhance data efficiency in large language models, achieving significant performance improvements on challenging tasks.
Why It Matters
The research addresses inefficiencies in traditional reinforcement learning methods, particularly in large language models. By improving data utilization and focusing on historically difficult samples, BAPO could lead to more effective AI systems, enhancing their reasoning capabilities and practical applications in various domains.
Key Takeaways
- BAPO improves data efficiency in large language models post-training.
- The framework dynamically selects training batches to enhance learning.
- BAPO shows a 12.5% performance improvement over existing methods.
- It resolves 40.7% of problems that base models fail to address.
- The findings could lead to advancements in AI reasoning capabilities.
Computer Science > Artificial Intelligence arXiv:2602.20722 (cs) [Submitted on 24 Feb 2026] Title:Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning Authors:Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun View a PDF of the paper titled Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning, by Xu Wan and 3 other authors View PDF HTML (experimental) Abstract:Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2602.20722 [cs.AI] (or arXiv:2602.20722v1 [cs.AI] for this version) https://doi.org/10.48550/arXiv.2602....
