[2603.22293] TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs

Computer Science > Computation and Language arXiv:2603.22293 (cs) [Submitted on 11 Mar 2026] Title:TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs Authors:Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Li Erran Li, Xiaolong Wang View a PDF of the paper titled TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs, by Yutao Xie and 5 other authors View PDF HTML (experimental) Abstract:Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate...