Related papers: Descent modulus and applications
This paper proposes a stochastic gradient descent method with an adaptive Gaussian noise term for the global minimization of nearly convex functions, which are nonconvex and possess multiple strict local minimizers. The noise term,…
A dynamical system is defined in terms of the gradient of a payoff function. Dynamical variables are of two types, ascent and descent. The ascent variables move in the direction of the gradient, while the descent variables move in the…
In the context of global optimization of mixed-integer nonlinear optimization formulations, we consider smoothing univariate functions $f$ that satisfy $f(0)=0$, $f$ is increasing and concave on $[0,+\infty)$, $f$ is twice differentiable on…
This article introduces a functional method for lower-dimensional smooth representations in terms of time-varying dissimilarities. The method incorporates dissimilarity representation in multidimensional scaling and smoothness approach of…
The Morse function $f$ near a non-degenerate critical point $p$ is understood topologically, in the light of Morse's lemma. However, Morse's lemma standardizes the function $f$ itself, providing little information of how the gradient…
Motivated by a wide variety of applications, ranging from stochastic optimization to dimension reduction through variable selection, the problem of estimating gradients accurately is of crucial importance in statistics and learning theory.…
For a locally Lipschitz continuous function $f:X\to\mathbb{R}$ the generalized gradient $\partial f(x)$ of Clarke is used to develop some (set-valued) gradient on a set $A\subset X$. Existence, uniqueness and some approximation are…
This work analyzes the convergence of a class of smoothing-based gradient descent methods when applied to optimization problems. In particular, Gaussian smoothing is employed to define a nonlocal gradient that reduces high-frequency noise,…
We present a stochastic descent algorithm for unconstrained optimization that is particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained optimization and…
We show that in a metric space, any continuous function with compact sublevel sets and finite metric slope is uniquely determined by the slope and its critical values.
In this paper we focus on the linear functionals defining an approximate version of the gradient of a function. These functionals are often used when dealing with optimization problems where the computation of the gradient of the objective…
In this paper, we provide near-optimal accelerated first-order methods for minimizing a broad class of smooth nonconvex functions that are strictly unimodal on all lines through a minimizer. This function class, which we call the class of…
Gradient-based iterative optimization methods are the workhorse of modern machine learning. They crucially rely on careful tuning of parameters like learning rate and momentum. However, one typically sets them using heuristic approaches…
The usual approach to developing and analyzing first-order methods for smooth convex optimization assumes that the gradient of the objective function is uniformly smooth with some Lipschitz constant $L$. However, in many settings the…
We study the generalization properties of unregularized gradient methods applied to separable linear classification -- a setting that has received considerable attention since the pioneering work of Soudry et al. (2018). We establish tight…
Let $\mathcal{A}$ denote the class of analytic functions $f$ on the unit disc $\mathbb{D}=\{z\in\mathbb{C}:\;|z|<1\}$ normalized by $f(0)=0$ and $f^{\prime}(0)=1$. In the present article, we consider and $\mathcal{F}(c)$ the subclasses of…
We consider solving nonconvex composite optimization problems in which the sum of a smooth function and a nonsmooth function is minimized. Many of convergence analyses of proximal gradient-type methods rely on global descent property…
We consider the problem of optimising the expected value of a loss functional over a nonlinear model class of functions, assuming that we have only access to realisations of the gradient of the loss. This is a classical task in statistics,…
It was established in [8] that Lipschitz inf-compact functions are uniquely determined by their local slope and critical values. Compactness played a paramount role in this result, ensuring in particular the existence of critical points. We…
Stochastic gradient descent (SGD) is widely used in machine learning. Although being commonly viewed as a fast but not accurate version of gradient descent (GD), it always finds better solutions than GD for modern neural networks. In order…