[2604.01613] Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error

Computer Science > Machine Learning arXiv:2604.01613 (cs) [Submitted on 2 Apr 2026] Title:Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error Authors:Taisuke Kobayashi View a PDF of the paper titled Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error, by Taisuke Kobayashi View PDF HTML (experimental) Abstract:In reinforcement learning (RL), temporal difference (TD) errors are widely adopted for optimizing value and policy functions. However, since the TD error is defined by a bootstrap method, its computation tends to be noisy and destabilize learning. Heuristics to improve the accuracy of TD errors, such as target networks and ensemble models, have been introduced so far. While these are essential approaches for the current deep RL algorithms, they cause side effects like increased computational cost and reduced learning efficiency. Therefore, this paper revisits the TD learning algorithm based on control as inference, deriving a novel algorithm capable of robust learning against noisy TD errors. First, the distribution model of optimality, a binary random variable, is represented by a sigmoid function. Alongside forward and reverse Kullback-Leibler divergences, this new model derives a robust learning rule: when the sigmoid function saturates with a large TD error probably due to noise, the gradient vanishes, implicitly excluding it from learning. Furthermore, the two divergences exhibit distinct...