![[2503.18980] CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2503.18980] CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning
Summary
The paper introduces CAE, a novel approach in deep reinforcement learning that repurposes value networks to enhance exploration efficiency without adding parameters, achieving strong empirical results.
Why It Matters
Exploration is a critical challenge in reinforcement learning, and CAE offers a theoretically sound and practical solution. By improving exploration strategies, this research has the potential to advance the effectiveness of RL applications across various domains.
Key Takeaways
- CAE repurposes value networks to enhance exploration in RL.
- The method requires minimal code changes, making it accessible for implementation.
- CAE+ extends CAE with an auxiliary network while maintaining simplicity.
- Empirical tests on multiple platforms validate CAE's effectiveness.
- The approach combines theoretical rigor with practical application.
Statistics > Machine Learning arXiv:2503.18980 (stat) [Submitted on 23 Mar 2025 (v1), last revised 20 Feb 2026 (this version, v2)] Title:CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning Authors:Yexin Li View a PDF of the paper titled CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning, by Yexin Li View PDF HTML (experimental) Abstract:Exploration remains a fundamental challenge in reinforcement learning, as many existing methods either lack theoretical guarantees or fall short in practical effectiveness. In this paper, we propose CAE, i.e., the Critic as an Explorer, a lightweight approach that repurposes the value networks in standard deep RL algorithms to drive exploration, without introducing additional parameters. CAE leverages multi-armed bandit techniques combined with a tailored scaling strategy, enabling efficient exploration with provable sub-linear regret bounds and strong empirical stability. Remarkably, it is simple to implement, requiring only about 10 lines of code. For complex tasks where learning reliable value networks is difficult, we introduce CAE+, an extension of CAE that incorporates an auxiliary network. CAE+ increases the parameter count by less than 1% while preserving implementation simplicity, adding roughly 10 additional lines of code. Extensive experiments on MuJoCo, MiniHack, and Habitat validate the effectiveness of CAE and CAE+, highlighting their ability to unify theoretical rigor with practica...
