![[2602.20132] LAD: Learning Advantage Distribution for Reasoning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20132] LAD: Learning Advantage Distribution for Reasoning
Summary
The paper introduces Learning Advantage Distribution (LAD), a novel framework for improving reasoning in reinforcement learning by focusing on advantage-induced distributions rather than solely maximizing rewards.
Why It Matters
This research addresses the limitations of traditional reinforcement learning methods that often lead to overfitting on dominant reward signals. By proposing LAD, the authors aim to enhance the diversity and accuracy of reasoning in large models, which is crucial for advancing AI capabilities in complex tasks.
Key Takeaways
- LAD framework improves reasoning by focusing on advantage distributions.
- It prevents overfitting to dominant reward signals, enhancing exploration.
- LAD incurs no extra training costs and scales well with existing models.
- Experimental results show improved accuracy and generative diversity.
- The approach is validated in controlled settings, demonstrating its effectiveness.
Computer Science > Machine Learning arXiv:2602.20132 (cs) [Submitted on 23 Feb 2026] Title:LAD: Learning Advantage Distribution for Reasoning Authors:Wendi Li, Sharon Li View a PDF of the paper titled LAD: Learning Advantage Distribution for Reasoning, by Wendi Li and 1 other authors View PDF HTML (experimental) Abstract:Current reinforcement learning objectives for large-model reasoning primarily focus on maximizing expected rewards. This paradigm can lead to overfitting to dominant reward signals, while neglecting alternative yet valid reasoning trajectories, thereby limiting diversity and exploration. To address this issue, we introduce Learning Advantage Distributions (LAD), a distribution-matching framework that replaces advantage maximization with learning the advantage-induced distribution. By establishing the equivalence between the optimal policy update and an advantage-based target distribution, we derive a practical LAD objective formulated as minimizing an $f$-divergence between the policy-induced and advantage-induced distributions. This yields a gradient update that increases likelihood for high-advantage responses while suppressing over-confident probability growth, preventing collapse without requiring auxiliary entropy regularization. LAD incurs no extra training cost compared to GRPO and scales naturally to LLM post-training. In a controlled bandit setting, LAD faithfully recovers the multimodal advantage distribution, validating the theoretical formulati...