![[2511.20718] Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2511.20718] Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Summary
This article presents SORL, a novel approach to stabilize off-policy training for long-horizon LLM agents, addressing issues of instability in reinforcement learning algorithms like PPO and GRPO.
Why It Matters
The findings are significant for researchers and practitioners in machine learning, particularly in reinforcement learning, as they tackle common challenges of instability and performance collapse in training large language models for multi-turn tasks. The proposed methods could enhance the reliability and efficiency of LLM training processes.
Key Takeaways
- SORL introduces mechanisms to stabilize off-policy training for LLM agents.
- The approach addresses granularity mismatches and high variance in gradient updates.
- SO-PPO and SO-GRPO algorithms show improved stability and performance in multi-turn tasks.
- Empirical results demonstrate reduced training instabilities compared to standard methods.
- The framework offers a scalable solution for reinforcement learning challenges.
Computer Science > Machine Learning arXiv:2511.20718 (cs) [Submitted on 25 Nov 2025 (v1), last revised 24 Feb 2026 (this version, v2)] Title:Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization Authors:Chenliang Li, Adel Elmahdy, Alex Boyd, Zhongruo Wang, Siliang Zeng, Alfredo Garcia, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Mingyi Hong View a PDF of the paper titled Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization, by Chenliang Li and 9 other authors View PDF HTML (experimental) Abstract:Reinforcement learning (RL) algorithms such as PPO and GRPO are widely used to train large language models (LLMs) for multi-turn agentic tasks. However, in off-policy training pipelines, these methods often exhibit unstable optimization dynamics and are prone to performance collapse. Through empirical analysis, we identify two fundamental sources of instability in this setting: (1)~a granularity mismatch between token-level policy optimization and turn-structured interactions, and (2) high-variance and unreliable gradient updates induced by off-policy importance sampling and inaccurate advantage estimation. To address these challenges, we propose SORL, \underline{S}tabilizing \underline{O}ff-Policy \underline{R}einforcement \underline{L}earning for Long-Horizon Agent Training. SORL introduces principled mechanisms that align pol...
