Related papers: Modified Equations for Stochastic Optimization
Gradient optimization algorithms using epochs, that is those based on stochastic gradient descent without replacement (SGDo), are predominantly used to train machine learning models in practice. However, the mathematical theory of SGDo and…
Recently, it has been shown in [Hairer, M., Hutzenthaler, M., Jentzen, A., Loss of regularity for Kolmogorov equations, Ann. Probab. 43, 2 (2015), 468--527] that there exists a system of stochastic differential equations (SDE) on the time…
We study pathwise approximation of scalar stochastic differential equations at a single point. We provide the exact rate of convergence of the minimal errors that can be achieved by arbitrary numerical methods that are based (in a…
We present a novel variational framework for performing inference in (neural) stochastic differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (fBM). SDEs offer a versatile tool for modeling real-world…
Inverse problems in scientific computing often require optimization over infinite-dimensional Hilbert spaces. A commonly used solver in such settings is stochastic gradient descent (SGD), where gradients are approximated using randomly…
We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with…
We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified…
Recently, it has been shown in [Jentzen, A., M\"uller-Gronbach, T., and Yaroslavtseva, L., Commun. Math. Sci., 14, 2016] that there exists a system of autonomous stochastic differential equations (SDE) on the time interval $[0,T]$ with…
Our subject of study is strong approximation of stochastic differential equations (SDEs) with respect to the supremum error criterion, and we seek approximations that are strongly asymptotically optimal in specific classes of…
We give a new take on the error analysis of approximations of stochastic differential equations (SDEs), utilizing and developing the stochastic sewing lemma of L\^e (2020). This approach allows one to exploit regularization by noise effects…
We propose a stochastic modified equations (SME) for modeling the asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the…
We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together…
We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective…
In this paper, we study numerical approximations for stochastic differential equations (SDEs) that use adaptive step sizes. In particular, we consider a general setting where decisions to reduce step sizes are allowed to depend on the…
The rate of strong convergence is investigated for an approximation scheme for a class of stochastic differential equations driven by a time-changed Brownian motion, where the random time changes $(E_t)_{t\ge 0}$ considered include the…
Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN)…
Stochastic differential equations (SDEs) provide a flexible framework for modeling temporal dynamics in partially observed systems. A central task is to calibrate such models from data, which requires inferring latent trajectories and…
Stochastic differential equations (SDEs) provide a natural framework for modelling intrinsic stochasticity inherent in many continuous-time physical processes. When such processes are observed in multiple individuals or experimental units,…
Stochastic symmetries and related invariance properties of finite dimensional SDEs driven by general cadlag semimartingales taking values in Lie groups are defined and investigated. The considered set of SDEs, first introduced by S. Cohen,…
In the recent article [Hairer, M., Hutzenthaler, M., Jentzen, A., Loss of regularity for Kolmogorov equations, Ann. Probab. 43 (2015), no. 2, 468--527] it has been shown that there exist stochastic differential equations (SDEs) with…