![[2312.02355] When is Offline Policy Selection Sample Efficient for Reinforcement Learning?](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2312.02355] When is Offline Policy Selection Sample Efficient for Reinforcement Learning?
Summary
This paper explores the efficiency of offline policy selection (OPS) in reinforcement learning, connecting it to off-policy evaluation (OPE) and Bellman error (BE) estimation, and introduces a new method called Identifiable BE Selection (IBES).
Why It Matters
Understanding the efficiency of offline policy selection is crucial for improving reinforcement learning algorithms, especially in environments where sample efficiency is limited. This research clarifies the relationship between OPS and OPE, providing insights that can enhance the development of more effective learning strategies.
Key Takeaways
- OPS is as hard as OPE in the worst case, limiting sample efficiency.
- Bellman error estimation can improve sample efficiency for OPS under certain conditions.
- The Identifiable BE Selection (IBES) method offers a new approach for selecting hyperparameters in OPS.
Computer Science > Machine Learning arXiv:2312.02355 (cs) [Submitted on 4 Dec 2023 (v1), last revised 15 Feb 2026 (this version, v2)] Title:When is Offline Policy Selection Sample Efficient for Reinforcement Learning? Authors:Vincent Liu, Prabhat Nagarajan, Andrew Patterson, Martha White View a PDF of the paper titled When is Offline Policy Selection Sample Efficient for Reinforcement Learning?, by Vincent Liu and 3 other authors View PDF HTML (experimental) Abstract:Offline reinforcement learning algorithms often require careful hyperparameter tuning. Before deployment, we need to select amongst a set of candidate policies. However, there is limited understanding about the fundamental limits of this offline policy selection (OPS) problem. In this work we provide clarity on when sample efficient OPS is possible, primarily by connecting OPS to off-policy policy evaluation (OPE) and Bellman error (BE) estimation. We first show a hardness result, that in the worst case, OPS is just as hard as OPE, by proving a reduction of OPE to OPS. As a result, no OPS method can be more sample efficient than OPE in the worst case. We then connect BE estimation to the OPS problem, showing how BE can be used as a tool for OPS. While BE-based methods generally require stronger requirements than OPE, when those conditions are met they can be more sample efficient. Building on this insight, we propose a BE method for OPS, called Identifiable BE Selection (IBES), that has a straightforward metho...
