Correcting Auto-Differentiation in Neural-ODE Training
Abstract
Does the use of auto-differentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by auto-differentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as Linear Multistep Methods (LMM) or Explicit Runge-Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force auto-differentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of Leapfrog and 2-stage ERK, we propose simple post-processing techniques that effectively eliminates these oscillations, correct the gradient computation and thus returns the accurate updates.
Cite
@article{arxiv.2306.02192,
title = {Correcting Auto-Differentiation in Neural-ODE Training},
author = {Yewei Xu and Shi Chen and Qin Li},
journal= {arXiv preprint arXiv:2306.02192},
year = {2026}
}
Comments
Accepted for publication in SIAM Journal on Applied Mathematics. This version corresponds to the final draft, prior to copyediting and production