[2604.01345] Malliavin Calculus for Counterfactual Gradient Estimation in Adaptive Inverse Reinforcement Learning

Computer Science > Machine Learning arXiv:2604.01345 (cs) [Submitted on 1 Apr 2026] Title:Malliavin Calculus for Counterfactual Gradient Estimation in Adaptive Inverse Reinforcement Learning Authors:Vikram Krishnamurthy, Luke Snow View a PDF of the paper titled Malliavin Calculus for Counterfactual Gradient Estimation in Adaptive Inverse Reinforcement Learning, by Vikram Krishnamurthy and 1 other authors View PDF HTML (experimental) Abstract:Inverse reinforcement learning (IRL) recovers the loss function of a forward learner from its observed responses adaptive IRL aims to reconstruct the loss function of a forward learner by passively observing its gradients as it performs reinforcement learning (RL). This paper proposes a novel passive Langevin-based algorithm that achieves adaptive IRL. The key difficulty in adaptive IRL is that the required gradients in the passive algorithm are counterfactual, that is, they are conditioned on events of probability zero under the forward learner's trajectory. Therefore, naive Monte Carlo estimators are prohibitively inefficient, and kernel smoothing, though common, suffers from slow convergence. We overcome this by employing Malliavin calculus to efficiently estimate the required counterfactual gradients. We reformulate the counterfactual conditioning as a ratio of unconditioned expectations involving Malliavin quantities, thus recovering standard estimation rates. We derive the necessary Malliavin derivatives and their adjoint Skoroho...