Related papers: One-dimensional System Arising in Stochastic Gradi…

We study a stochastic process $X_t$ related to the Bessel and the Rayleigh processes, with various applications in physics, chemistry, biology, economics, finance and other fields. The stochastic differential equation is $dX_t = (nD/X_t) dt…

Optimization algorithms are unlikely to converge to strict saddle points. Proofs to that effect rely on the Center-Stable Manifold Theorem (CSMT), casting algorithms as dynamical systems: $x_{k+1} = g_k(x_k)$. In its standard form, the CSMT…

In this paper, given two Banach spaces $X, Y$ and a $C^1$ functional $\Phi:X\times Y\to {\bf R}$, under general assumptions, we show that either $\Phi$ has a saddle-point in $X\times Y$ or, for each convex and dense set $S\subseteq Y$,…

Without randomization, escaping the saddle points of $f \colon \mathbb{R}^d \to \mathbb{R}$ requires at least $\Omega(d)$ pieces of information about $f$ (values, gradients, Hessian-vector products). With randomization, this can be reduced…

Let $X$ be a $d$-dimensional Gaussian process in $[0,1]$, where the component are independent copies of a scalar Gaussian process $X_0$ on $[0,1]$ with a given general variance function $\gamma^2(r)=\operatorname{Var}\left(X_0(r)\right)$…

The note considers normalized gradient descent (NGD), a natural modification of classical gradient descent (GD) in optimization problems. A serious shortcoming of GD in non-convex problems is that GD may take arbitrarily long to escape from…

It is known that step size adaptive evolution strategies (ES) do not converge (prematurely) to regular points of continuously differentiable objective functions. Among critical points, convergence to minima is desired, and convergence to…

We present a comprehensive theoretical analysis of first-order methods for escaping strict saddle points in smooth non-convex optimization. Our main contribution is a Perturbed Saddle-escape Descent (PSD) algorithm with fully explicit…

We study the problem of differentially-private (DP) stochastic (convex-concave) saddle-points in the $\ell_1$ setting. We propose $(\varepsilon, \delta)$-DP algorithms based on stochastic mirror descent that attain nearly…

We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show…

We consider the stochastic differential equation $dX_t = A(X_{t-}) \, dZ_t$, $ X_0 = x$, driven by cylindrical $\alpha$-stable process $Z_t$ in $R^d$, where $\alpha \in (0,1)$ and $d \ge 2$. We assume that the determinant of $A(x) =…

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points.…

We analyze stochastic gradient algorithms for optimizing nonconvex problems. In particular, our goal is to find local minima (second-order stationary points) instead of just finding first-order stationary points which may be some bad…

A saddlepoint approximation of the Student's t-statistic was derived by Daniels and Young [Biometrika 78 (1991) 169-179] under the very stringent exponential moment condition that requires that the underlying density function go down at…

Recently, it has been shown in [Jentzen, A., M\"uller-Gronbach, T., and Yaroslavtseva, L., Commun. Math. Sci., 14, 2016] that there exists a system of autonomous stochastic differential equations (SDE) on the time interval $[0,T]$ with…

In a series of papers \cite{LSJR16, PP17, LPP}, it was established that some of the most commonly used first order methods almost surely (under random initializations) and with step-size being small enough, avoid strict saddle points, as…

The classical analysis of Stochastic Gradient Descent (SGD) with polynomially decaying stepsize $\eta_t = \eta/\sqrt{t}$ relies on well-tuned $\eta$ depending on problem parameters such as Lipschitz smoothness constant, which is often…

Stochastic gradient descent (SGD) has been widely used in machine learning due to its computational efficiency and favorable generalization properties. Recently, it has been empirically demonstrated that the gradient noise in several deep…

We investigate three types of averaging principles and the normal deviation for multi-scale stochastic differential equations (in short, SDEs) with polynomial nonlinearity. More specifically, we first demonstrate the strong convergence of…

We develop a new continuous-time stochastic gradient descent method for optimizing over the stationary distribution of stochastic differential equation (SDE) models. The algorithm continuously updates the SDE model's parameters using an…