Related papers: Gradients on Sets
We consider the problem of minimizing a convex function that depends on an uncertain parameter $\theta$. The uncertainty in the objective function means that the optimum, $x^*(\theta)$, is also a function of $\theta$. We propose an…
We develop a distributed stochastic gradient descent algorithm for solving non-convex optimization problems under the assumption that the local objective functions are twice continuously differentiable with Lipschitz continuous gradients…
A vast literature on convergence guarantees for gradient descent and derived methods exists at the moment. However, a simple practical situation remains unexplored: when a fixed step size is used, can we expect gradient descent to converge…
In this paper we consider large-scale composite nonconvex optimization problems having the objective function formed as a sum of three terms, first has block coordinate-wise Lipschitz continuous gradient, second is twice differentiable but…
We introduce a notion of inexact model of a convex objective function, which allows for errors both in the function and in its gradient. For this situation, a gradient method with an adaptive adjustment of some parameters of the model is…
We consider the problem of optimising the expected value of a loss functional over a nonlinear model class of functions, assuming that we have only access to realisations of the gradient of the loss. This is a classical task in statistics,…
For strongly convex objectives that are smooth, the classical theory of gradient descent ensures linear convergence relative to the number of gradient evaluations. An analogous nonsmooth theory is challenging. Even when the objective is…
An open question in the Deep Learning community is why neural networks trained with Gradient Descent generalize well on real datasets even though they are capable of fitting random data. We propose an approach to answering this question…
We prove that given a locally integrable function $f$ on an open set of an Euclidean space the distributional derivative $Xf$ with respect to a locally Lipshitzian vector field $X$ is locally integrable if, and only if, the function $f$…
In this paper we propose several adaptive gradient methods for stochastic optimization. Unlike AdaGrad-type of methods, our algorithms are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of…
Recent studies have shown that many nonconvex machine learning problems satisfy a generalized-smooth condition that extends beyond traditional smooth nonconvex optimization. However, the existing algorithms are not fully adapted to such…
We study the convergence properties of gradient descent for training deep linear neural networks, i.e., deep matrix factorizations, by extending a previous analysis for the related gradient flow. We show that under suitable conditions on…
We consider a composite optimization problem where the sum of a continuously differentiable and a merely lower semicontinuous function has to be minimized. The proximal gradient algorithm is the classical method for solving such a problem…
This paper considers optimization of smooth nonconvex functionals in smooth infinite dimensional spaces. A H\"older gradient descent algorithm is first proposed for finding approximate first-order points of regularized polynomial…
This paper deals with two kinds of the one-dimensional global optimization problems over a closed finite interval: (i) the objective function $f(x)$ satisfies the Lipschitz condition with a constant $L$; (ii) the first derivative of $f(x)$…
The gradient descent method aims at finding local minima of a given multivariate function by moving along the direction of its gradient, and hence, the algorithm typically involves computing all partial derivatives of a given function,…
We revisit the geometrically decaying step size given a positive inverse condition number, under which a locally Lipschitz function shows linear convergence. The positivity does not require the function to satisfy convexity, weak convexity,…
Randomized smoothing is a widely adopted technique for optimizing nonsmooth objective functions. However, its efficiency analysis typically relies on global Lipschitz continuity, a condition rarely met in practical applications. To address…
This paper addresses the unconstrained minimization of smooth convex functions whose gradients are locally Holder continuous. Building on these results, we analyze the Scaled Gradient Algorithm (SGA) under local smoothness assumptions,…
Stochastic gradient descent (SGD) has been a go-to algorithm for nonconvex stochastic optimization problems arising in machine learning. Its theory however often requires a strong framework to guarantee convergence properties. We hereby…