Related papers: Potential-Function Proofs for First-Order Methods
First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent,…
Constrained optimization problems where both the objective and constraints may be nonsmooth and nonconvex arise across many learning and data science settings. In this paper, we show for any Lipschitz, weakly convex objectives and…
Arising in semi-parametric statistics, control applications, and as sub-problems in global optimization methods, certain optimization problems can have objective functions requiring numerical integration to evaluate, yet gradient function…
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a…
We present a new feasible proximal gradient method for constrained optimization where both the objective and constraint functions are given by the summation of a smooth, possibly nonconvex function and a convex simple function. The…
Polyak-{\L}ojasiewicz (PL) [Polyak, 1963] condition is a weaker condition than the strong convexity but suffices to ensure a global convergence for the Gradient Descent algorithm. In this paper, we study the lower bound of algorithms using…
Learning new tasks by drawing on prior experience gathered from other (related) tasks is a core property of any intelligent system. Gradient-based meta-learning, especially MAML and its variants, has emerged as a viable solution to…
We study the optimization of non-convex functions that are not necessarily smooth (gradient and/or Hessian are Lipschitz) using first order methods. Smoothness is a restrictive assumption in machine learning in both theory and practice,…
We focus on nonconvex and nonsmooth minimization problems with a composite objective, where the differentiable part of the objective is freed from the usual and restrictive global Lipschitz gradient continuity assumption. This longstanding…
This paper provides a theoretical and numerical comparison of classical first-order splitting methods for solving smooth convex optimization problems and cocoercive equations. From a theoretical point of view, we compare convergence rates…
We introduce a numerical framework to verify the finite step convergence of first-order methods for parametric convex quadratic optimization. We formulate the verification problem as a mathematical optimization problem where we maximize a…
We propose a new methodology to design first-order methods for unconstrained strongly convex problems. Specifically, instead of tackling the original objective directly, we construct a shifted objective function that has the same minimizer…
We provide a novel computer-assisted technique for systematically analyzing first-order methods for optimization. In contrast with previous works, the approach is particularly suited for handling sublinear convergence rates and stochastic…
Classically, the time complexity of a first-order method is estimated by its number of gradient computations. In this paper, we study a more refined complexity by taking into account the `lingering' of gradients: once a gradient is computed…
This paper is devoted to studying the first-order variational analysis of non-convex and non-differentiable functions that may not be subdifferentially regular. To achieve this goal, we entirely rely on two concepts of directional…
We consider in this paper a class of single-ratio fractional minimization problems, in which the numerator part of the objective is the sum of a nonsmooth nonconvex function and a smooth nonconvex function while the denominator part is a…
We consider the problem of maximizing a convex function over a closed convex set in a real Hilbert space. For linear functions, we show that a single orthogonal projection suffices to obtain an approximate solution. For continuous convex…
We develop a theory of accelerated first-order optimization from the viewpoint of differential equations and Lyapunov functions. Building upon the previous work of many researchers, we consider differential equations which model the…
Making the gradients small is a fundamental optimization problem that has eluded unifying and simple convergence arguments in first-order optimization, so far primarily reserved for other convergence criteria, such as reducing the…
First-order methods for minimization and saddle point (min-max) problems are widely used for solving large-scale problems, in particular arising in machine learning. The majority of works obtain favorable complexity guarantees of such…