[2603.22713] Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Role of Bellman Constraints

Computer Science > Machine Learning arXiv:2603.22713 (cs) [Submitted on 24 Mar 2026] Title:Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Role of Bellman Constraints Authors:Tian Xu, Chenyang Wang, Xiaochen Zhai, Ziniu Li, Yi-Chen Li, Yang Yu View a PDF of the paper titled Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Role of Bellman Constraints, by Tian Xu and 5 other authors View PDF Abstract:Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors in behavioral cloning (BC), but often exhibits training instability due to adversarial optimization. To avoid this issue, a class of non-adversarial Q-based imitation learning (IL) methods, represented by IQ-Learn, has emerged and is widely believed to outperform BC by leveraging online environment interactions. However, this paper revisits IQ-Learn and demonstrates that it provably reduces to BC and suffers from an imitation gap lower bound with quadratic dependence on horizon, therefore still suffering from compounding errors. Theoretical analysis reveals that, despite using online interactions, IQ-Learn uniformly suppresses the Q-values for all actions on states uncovered by demonstrations, thereby failing to generalize. To address this limitation, we introduce a primal-dual framework for distribution matching, yielding a new Q-based IL method, Dual Q-DM. The key mechanism in Dual Q-DM is incorporating Bellman constraints...