[2603.23889] Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration

Computer Science > Machine Learning arXiv:2603.23889 (cs) [Submitted on 25 Mar 2026] Title:Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration Authors:Guopeng Li, Matthijs T.J. Spaan, Julian F.P. Kooij View a PDF of the paper titled Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration, by Guopeng Li and 2 other authors View PDF HTML (experimental) Abstract:When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offering high sample efficiency, suffer from constraint violations due to cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative offline distributional value learning. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile critics to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide exploration. Experiments on safe velocity, safe navigation, and autonomous driving tasks demonstrate that COX-Q ach...