[2603.19835] FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

Computer Science > Machine Learning arXiv:2603.19835 (cs) [Submitted on 20 Mar 2026] Title:FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization Authors:Chiyu Ma, Shuo Yang, Kexin Huang, Jinda Lu, Haoming Meng, Shangshang Wang, Bolin Ding, Soroush Vosoughi, Guoyin Wang, Jingren Zhou View a PDF of the paper titled FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization, by Chiyu Ma and 9 other authors View PDF HTML (experimental) Abstract:We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at ...