[2604.01597] Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training

Computer Science > Machine Learning arXiv:2604.01597 (cs) [Submitted on 2 Apr 2026] Title:Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training Authors:Dong Shu, Denghui Zhang, Jessica Hullman View a PDF of the paper titled Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training, by Dong Shu and 2 other authors View PDF HTML (experimental) Abstract:Traditional RL algorithms like Proximal Policy Optimization (PPO) typically train on the entire rollout buffer, operating under the assumption that all generated episodes provide a beneficial optimization signal. However, these episodes frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training. In this paper, we propose \textbf{Influence-Guided PPO (I-PPO)}, a novel framework that integrates data attribution into the RL post-training loop. By calculating an influence score for each episode using a gradient-based approximation, I-PPO identifies and eliminates episodes that are anti-aligned with a validation gradient. Our experiments demonstrate that I-PPO consistently outperforms SFT and PPO baselines. We show that our filtering process acts as an intrinsic early stopping mechanism, accelerating training efficiency while effectively reducing unfaithful CoT reasoning. Subjects: Machine Learning (cs.LG) Cite as: arXiv:2604.01597 [cs.LG]   (or arXiv:2604.01597v1 [cs.LG] for this version)   https://doi.org/10.48550/arXiv.2604.01...