Related papers: Exponential convergence of Sobolev gradient descen…
The homogeneous second-order descent method (Zhang et al. 2025, Mathematics of Operations Research) was initially proposed for unconstrained optimisation problems. HSODM shows excellent performance with respect to the global complexity rate…
We show that the joint spectral radius of a finite collection of nonnegative matrices can be bounded by the eigenvalue of a non-linear operator. This eigenvalue coincides with the ergodic constant of a risk-sensitive control problem, or of…
We analyze the complexity of biased stochastic gradient methods (SGD), where individual updates are corrupted by deterministic, i.e. biased error terms. We derive convergence results for smooth (non-convex) functions and give improved rates…
Nonconvex and nonsmooth optimization problems are important and challenging for statistics and machine learning. In this paper, we propose Projected Proximal Gradient Descent (PPGD) which solves a class of nonconvex and nonsmooth…
We study nonexistence results and gradient estimates for solutions of \[ \Delta_p v + a v^{q}=0 \] defined on complete Riemannian manifolds satisfying a \emph{$\chi$-type Sobolev inequality}. We establish a Liouville theorem under the…
Due to simplicity, computational cheapness, and efficiency, the Barzilai and Borwein (BB) gradient method has received a significant amount of attention in different fields of optimization. In the first part of this paper, based on spectral…
This paper develops expansive gradient dynamics in deep neural network-induced mapping spaces. Specifically, we generate tools and concepts for minimizing a class of energy functionals in an abstract Hilbert space setting covering a wide…
The subgradient method is a classical and foundational approach in non-smooth convex optimization; its simplicity, robustness, and role as a conceptual and algorithmic starting point have made it the backbone of many significant…
Stochastic gradient descent is one of the most successful approaches for solving large-scale problems, especially in machine learning and statistics. At each iteration, it employs an unbiased estimator of the full gradient computed from one…
We analyze the constant step size subgradient method on nonsmooth, nonconvex functions. We identify geometric assumptions on the objective function under which i) its domain admits a partition (stratification) into smooth manifolds (strata)…
We consider a variable metric linesearch based proximal gradient method for the minimization of the sum of a smooth, possibly nonconvex function plus a convex, possibly nonsmooth term. We prove convergence of this iterative algorithm to a…
In this work, we establish that Nesterov's accelerated gradient method, applied to $C^2$ functions satisfying the Polyak--{\L}ojasiewicz inequality around local minimizers, achieves the optimal local linear convergence rate…
Stochastic gradient descent is one of the most common iterative algorithms used in machine learning and its convergence analysis is a rich area of research. Understanding its convergence properties can help inform what modifications of it…
We analyze backward step control globalization for finding zeros of G\^ateaux-differentiable functions that map from a Banach space to a Hilbert space. The results include global convergence to a distinctive solution characterized by…
Most prior work on the convergence of gradient descent (GD) for overparameterized neural networks relies on strong assumptions on the step size (infinitesimal), the hidden-layer width (infinite), or the initialization (large, spectral,…
Adaptive gradient-descent optimizers are the standard choice for training neural network models. Despite their faster convergence than gradient-descent and remarkable performance in practice, the adaptive optimizers are not as well…
Anderson acceleration (or Anderson mixing) is an efficient acceleration method for fixed point iterations $x_{t+1}=G(x_t)$, e.g., gradient descent can be viewed as iteratively applying the operation $G(x) \triangleq x-\alpha\nabla f(x)$. It…
Proximal gradient methods are a popular tool for the solution of structured, nonsmooth minimization problems. In this work, we investigate an extension of the former to general Banach spaces and provide worst-case convergence rates for,…
Gradient descent is a simple and widely used optimization method for machine learning. For homogeneous linear classifiers applied to separable data, gradient descent has been shown to converge to the maximal margin (or equivalently, the…
In this paper, we propose a random gradient-free method for optimization in infinite dimensional Hilbert spaces, applicable to functional optimization in diverse settings. Though such problems are often solved through finite-dimensional…