Related papers: Modified Equations for Stochastic Optimization
Stochastic gradient descent (SGD) is perhaps the most prevalent optimization method in modern machine learning. Contrary to the empirical practice of sampling from the datasets without replacement and with (possible) reshuffling at each…
Probabilistic ordinary differential equation (ODE) solvers have been introduced over the past decade as uncertainty-aware numerical integrators. They typically proceed by assuming a functional prior to the ODE solution, which is then…
The aim of this work is to provide the strong convergence results of numerical approximations of a general second order non-autonomous semilinear stochastic partial differential equation (SPDE) driven simultaneously by an additive…
Exponentiated gradient descent (EGD), a biologically motivated optimisation algorithm that respects Dale's law, produces log-normally distributed synaptic weights at convergence, in alignment with experimental observations in neuroscience.…
For a stochastic differential equation(SDE) driven by a fractional Brownian motion(fBm) with Hurst parameter $H>\frac{1}{2}$, it is known that the existing (naive) Euler scheme has the rate of convergence $n^{1-2H}$. Since the limit…
We study the ergodic properties of finite-dimensional systems of SDEs driven by non-degenerate additive fractional Brownian motion with arbitrary Hurst parameter $H\in(0,1)$. A general framework is constructed to make precise the notions of…
Motivated by problems from statistical analysis for discretely sampled SPDEs, first we derive central limit theorems for higher order finite differences applied to stochastic process with arbitrary finitely regular paths. These results are…
In this paper, we investigate the problem of strong approximation of the solutions of stochastic differential equations (SDEs) when the drift coefficient is given in integral form. We investigate its upper error bounds, in terms of the…
We address a general optimal switching problem over finite horizon for a stochastic system described by a differential equation driven by Brownian motion. The main novelty is the fact that we allow for infinitely many modes (or regimes,…
This paper is devoted to the study of hyperbolic systems of linear partial differential equations perturbed by a Brownian motion. The existence and uniqueness of solutions are proved by an energy method. The specific features of this class…
A new notion of stochastic transformation is proposed and applied to the study of both weak and strong symmetries of stochastic differential equations (SDEs). The correspondence between an algebra of weak symmetries for a given SDE and an…
We discover restrained numerical instabilities in current training practices of deep networks with stochastic gradient descent (SGD), and its variants. We show numerical error (on the order of the smallest floating point bit and thus the…
We propose a new stochastic optimization framework for empirical risk minimization problems such as those that arise in machine learning. The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an…
In the past decade, an intensive study of strong approximation of stochastic differential equations (SDEs) with a drift coefficient that has discontinuities in space has begun. In the majority of these results it is assumed that the drift…
This thesis is devoted to the study of extreme value statistics in stochastic processes and their applications. In the first part, we obtain exact analytical results on the extreme value statistics of both discrete-time and continuous-time…
This paper deals with asymptotic errors, limit theorems for errors between numerical and exact solutions of stochastic differential equation (SDE) driven by one-dimensional fractional Brownian motion (fBm). The Euler-Maruyama, higher-order…
In this thesis, we propose new theoretical frameworks for the analysis of stochastic and distributed methods with error compensation and local updates. Using these frameworks, we develop more than 20 new optimization methods, including the…
Minimax optimization problems have attracted a lot of attention over the past few years, with applications ranging from economics to machine learning. While advanced optimization methods exist for such problems, characterizing their…
Our approach is part of the close link between continuous dissipative dynamical systems and optimization algorithms. We aim to solve convex minimization problems by means of stochastic inertial differential equations which are driven by the…
We investigate synchronization by noise for stochastic differential equations (SDEs) driven by a fractional Brownian motion (fbm) with Hurst index $H\in(0,1)$. Provided that the SDE has a negative top Lyapunov exponent, we show that a weak…