[2603.20046] Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs

Computer Science > Artificial Intelligence arXiv:2603.20046 (cs) [Submitted on 20 Mar 2026] Title:Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs Authors:Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi, Baisheng Lai, Jianqiang Huang View a PDF of the paper titled Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs, by Wenjian Zhang and 4 other authors View PDF Abstract:Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to curent policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learnin...