[2603.22273] Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration

Computer Science > Machine Learning arXiv:2603.22273 (cs) [Submitted on 23 Mar 2026] Title:Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration Authors:Zakaria Mhammedi, James Cohan View a PDF of the paper titled Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration, by Zakaria Mhammedi and 1 other authors View PDF Abstract:The process of discovery requires active exploration -- the act of collecting new and informative data. However, efficient autonomous exploration remains a major unsolved problem. The dominant paradigm addresses this challenge by using Reinforcement Learning (RL) to train agents with intrinsic motivation, maximizing a composite objective of extrinsic and intrinsic rewards. We suggest that this approach incurs unnecessary overhead: while policy optimization is necessary for precise task execution, employing such machinery solely to expand state coverage may be inefficient. In this paper, we propose a new paradigm that explicitly separates exploration from exploitation and bypasses RL during the exploration phase. Our method uses a tree-search strategy inspired by the Go-With-The-Winner algorithm, paired with a measure of epistemic uncertainty to systematically drive exploration. By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard Atari benchmarks. Furthe...