Related papers: Modified Equations for Stochastic Optimization

Stochastic Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion. However, this approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not…

We study the scaling limits of stochastic gradient descent (SGD) with constant step-size in the high-dimensional regime. We prove limit theorems for the trajectories of summary statistics (i.e., finite-dimensional functions) of SGD as the…

We study the discrete-time approximation for solutions of quadratic forward back- ward stochastic differential equations (FBSDEs) driven by a Brownian motion and a jump process which could be dependent. Assuming that the generator has a…

We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix…

The choice of how to retain information about past gradients dramatically affects the convergence properties of state-of-the-art stochastic optimization methods, such as Heavy-ball, Nesterov's momentum, RMSprop and Adam. Building on this…

A theoretical, and potentially also practical, problem with stochastic gradient descent is that trajectories may escape to infinity. In this note, we investigate uniform boundedness properties of iterates and function values along the…

We study the approximation of the ergodic measure of the following stochastic differential equation (SDE) on $\mathbb{R}^d$: \begin{eqnarray}\label{e:SDEE} d X_t &=& (b_1(X_t)+b_2(X_t)) d t+\sigma(X_t) d W_t, \end{eqnarray} where $W_t$ is a…

We investigate the validity and accuracy of weak-noise (saddle-point or instanton) approximations for piecewise-smooth stochastic differential equations (SDEs), taking as an illustrative example a piecewise-constant SDE, which serves as a…

We study the convergence of a generic tamed Euler-Maruyama (EM) scheme for the kinetic type stochastic differential equations (SDEs) (also known as second order SDEs) with singular coefficients in both weak and strong probabilistic senses.…

We present an optimizer which uses Bayesian optimization to tune the system parameters of distributed stochastic gradient descent (SGD). Given a specific context, our goal is to quickly find efficient configurations which appropriately…

The paper focuses on discrete-type approximations of solutions to non-homogeneous stochastic differential equations (SDEs) involving fractional Brownian motion (fBm). We prove that the rate of convergence for Euler approximations of…

Despite the success of adaptive time-stepping in ODE simulation, it has so far seen few applications for Stochastic Differential Equations (SDEs). To simulate SDEs adaptively, methods such as the Virtual Brownian Tree (VBT) have been…

We study sequences of empirical measures of Euler schemes associated to some non-Markovian SDEs: SDEs driven by Gaussian processes with stationary increments. We obtain the functional convergence of this sequence to a stationary solution to…

In this paper we analyze the behaviour of the stochastic gradient descent (SGD), a widely used method in supervised learning for optimizing neural network weights via a minimization of non-convex loss functions. Since the pioneering work of…

Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due…

Conditional stochastic optimization covers a variety of applications ranging from invariant learning and causal inference to meta-learning. However, constructing unbiased gradient estimators for such problems is challenging due to the…

We consider the problem of the simulation of Levy-driven stochastic differential equations. It is generally impossible to simulate the increments of a Levy-process. Thus in addition to an Euler scheme, we have to simulate approximately…

We study geometric stochastic differential equations (SDEs) and their approximations on Riemannian manifolds. In particular, we introduce a simple new construction of geometric SDEs, using which with bounded curvature. In particular, we…

Learning unknown stochastic differential equations (SDEs) from observed data is a significant and challenging task with applications in various fields. Current approaches often use neural networks to represent drift and diffusion functions,…

Stochastic differential equation mixed-effects models (SDEMEMs) are flexible hierarchical models that are able to account for random variability inherent in the underlying time-dynamics, as well as the variability between experimental units…