Related papers: Stochastic Modified Flows, Mean-Field Limits and D…
This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a two-layer neural network trained on Gaussian data and labels generated by a similar, though not necessarily identical, target function. We rigorously…
Motivated by the modeling of three-dimensional fluid turbulence, we define and study a class of stochastic partial differential equations (SPDEs) that are randomly stirred by a spatially smooth and uncorrelated in time forcing term. To…
Stochastic differential equations (SDEs) on compact foliated spaces were introduced a few years ago. As a corollary, a leafwise Brownian motion on a compact foliated space was obtained as a solution to an SDE. In this paper we construct…
By using a change of scale and space, we study a class of stochastic differential equations (SDEs) whose solutions are drift--perturbed and exhibit behaviour analogous to standard Brownian motion including to the Law of the Iterated…
In the vanishing learning rate regime, stochastic gradient descent (SGD) is now relatively well understood. In this work, we propose to study the basic properties of SGD and its variants in the non-vanishing learning rate regime. The focus…
We study the implicit regularization of mini-batch stochastic gradient descent, when applied to the fundamental problem of least squares regression. We leverage a continuous-time stochastic differential equation having the same moments as…
We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together…
In this paper, we study small-time asymptotic behaviors for a class of distribution dependent stochastic differential equations driven by fractional Brownian motions with Hurst parameter $H\in(1/2,1)$ and magnitude $\ep^H$. By building up a…
We show that the behavior of stochastic gradient descent is related to Bayesian statistics by showing that SGD is effectively diffusion on a fractal landscape, where the fractal dimension can be accounted for in a purely Bayesian way. By…
This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning…
We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing the difference…
We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with…
Motivated by the probabilistic representation for solutions of the Navier-Stokes equations, we introduce a novel class of stochastic differential equations that depend on the entire flow of its time marginals. We establish the existence and…
We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from…
In this paper we analyze the behaviour of the stochastic gradient descent (SGD), a widely used method in supervised learning for optimizing neural network weights via a minimization of non-convex loss functions. Since the pioneering work of…
For optimizing a non-convex function in finite dimension, a method is to add Brownian noise to a gradient descent, allowing for transitions between basins of attractions of different minimizers. To adapt this for optimization over a space…
We study the problem of optimal control for mean-field stochastic partial differential equations (stochastic evolution equations) driven by a Brownian motion and an independent Poisson random measure, in the case of \textit{partial…
We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance.…
In this paper we prove, for small Hurst parameters, the higher order differentiability of a stochastic flow associated with a stochastic differential equation driven by an additive multi-dimensional fractional Brownian noise, where the…
Deterministic flow models, such as rectified flows, offer a general framework for learning a deterministic transport map between two distributions, realized as the vector field for an ordinary differential equation (ODE). However, they are…