[2306.02192] Correcting Auto-Differentiation in Neural-ODE Training

Computer Science > Machine Learning arXiv:2306.02192 (cs) [Submitted on 3 Jun 2023 (v1), last revised 27 Mar 2026 (this version, v3)] Title:Correcting Auto-Differentiation in Neural-ODE Training Authors:Yewei Xu, Shi Chen, Qin Li View a PDF of the paper titled Correcting Auto-Differentiation in Neural-ODE Training, by Yewei Xu and 2 other authors View PDF HTML (experimental) Abstract:Does the use of auto-differentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by auto-differentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as Linear Multistep Methods (LMM) or Explicit Runge-Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force auto-differentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of Leapfrog and 2-stage ERK, we propose simple post-processing techniques that effectively eliminates these oscillations, correct the gradient computation and thus returns the accurate updates. Comments: Subjects: Machine Learning (cs.LG); Numerical Analysis (math.NA) MSC classes: 65D25 (Primary), 65L06, 90C31 (Secondary) Cite as: arXiv:2306.02192 [cs.LG]   (or arXiv:2306.02192v3 [cs.LG] for this version)   https://doi.org/10.48550/arXiv.2306.02192 Focus to learn more arXiv-issued DOI via DataCite Submi...