[2512.23805] Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting

Statistics > Machine Learning arXiv:2512.23805 (stat) [Submitted on 29 Dec 2025 (v1), last revised 21 Apr 2026 (this version, v2)] Title:Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting Authors:Lars van der Laan, Nathan Kallus View a PDF of the paper titled Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting, by Lars van der Laan and Nathan Kallus View PDF HTML (experimental) Abstract:Fitted Q-evaluation (FQE) is a foundational method for off-policy evaluation in reinforcement learning, but existing theory typically relies on Bellman completeness of the function class, a condition often violated in practice. This reliance is due to a fundamental norm mismatch: the Bellman operator is gamma-contractive in the L^2 norm induced by the target policy's stationary distribution, whereas standard FQE fits Bellman regressions under the behavior distribution. To resolve this mismatch, we reweight each Bellman regression step by an estimate of the stationary density ratio, inspired by emphatic weighting in temporal-difference learning. This makes the update behave as if it were performed under the target stationary distribution, restoring contraction without Bellman completeness while preserving the simplicity of regression-based evaluation. Illustrative experiments, including Baird's classical counterexample, show that stationary weighting can stabilize FQE under off-policy sampling. Subjects: Machine Learning (stat.ML); Machine Learnin...