![[2508.05612] Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2508.05612] Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
Summary
The paper presents Shuffle-R1, a novel reinforcement learning framework designed to enhance the efficiency of multimodal large language models by addressing training inefficiencies through dynamic trajectory sampling and batch reshuffling.
Why It Matters
As multimodal large language models (MLLMs) become increasingly prevalent, optimizing their training processes is crucial for improving their reasoning capabilities. This research addresses significant challenges in reinforcement learning, potentially leading to more effective and efficient model training, which can have wide-ranging implications for AI applications.
Key Takeaways
- Shuffle-R1 improves reinforcement learning efficiency for MLLMs.
- Addresses issues of Advantage Collapsing and Rollout Silencing.
- Introduces Pairwise Trajectory Sampling for better gradient signals.
- Advantage-based Trajectory Shuffle enhances valuable rollout exposure.
- Experimental results show consistent performance improvements over existing RL baselines.
Computer Science > Machine Learning arXiv:2508.05612 (cs) [Submitted on 7 Aug 2025 (v1), last revised 23 Feb 2026 (this version, v5)] Title:Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle Authors:Linghao Zhu, Yiran Guan, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Bin Qin, Jian Luan, Yuliang Liu, Xiang Bai View a PDF of the paper titled Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle, by Linghao Zhu and 8 other authors View PDF HTML (experimental) Abstract:Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM). However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing, where most advantages in a batch concentrate near zero, and Rollout Silencing, where the proportion of rollouts contributing non-zero gradients diminishes over time. These issues lead to suboptimal gradient updates and hinder long-term learning efficiency. To address these issues, we propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition. It introduces (1) Pairwise Trajectory Sampling, which selects high-contrast trajectories with large advantages to improve gradient signal quality, and (2) Advantage-based Trajector...
