[2006.04363] Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models

Computer Science > Machine Learning arXiv:2006.04363 (cs) [Submitted on 8 Jun 2020 (v1), last revised 3 Apr 2026 (this version, v2)] Title:Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models Authors:Farzane Aminmansour, Taher Jafferjee, Ehsan Imani, Erin Talvitie, Micheal Bowling, Martha White View a PDF of the paper titled Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models, by Farzane Aminmansour and 5 other authors View PDF HTML (experimental) Abstract:Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model. However, it is often difficult to learn accurate models of environment dynamics, and even small errors may result in failure of Dyna agents. In this paper, we highlight that one potential cause of that failure is bootstrapping off of the values of simulated states, and introduce a new Dyna algorithm to avoid this failure. We discuss a design space of Dyna algorithms, based on using successor or predecessor models -- simulating forwards or backwards -- and using one-step or multi-step updates. Three of the variants have been explored, but surprisingly the fourth variant has not: using predecessor models with multi-step updates. We present the \emph{Hallucinated Value Hypothesis} (HVH): updating the values of real states towards values of simulated states can result in misleading ...