Related papers: Whiplash Gradient Descent Dynamics
This paper considers the analysis of continuous time gradient-based optimization algorithms through the lens of nonlinear contraction theory. It demonstrates that in the case of a time-invariant objective, most elementary results on…
Recent focus on robustness to adversarial attacks for deep neural networks produced a large variety of algorithms for training robust models. Most of the effective algorithms involve solving the min-max optimization problem for training…
In this paper, we first study nonsmooth steepest descent method for nonsmooth functions defined on Hilbert space and establish the corresponding algorithm by proximal subgradients. Then, we use this algorithm to find stationary points for…
We study the Hamiltonian flow for optimization (HF-opt), which simulates the Hamiltonian dynamics for some integration time and resets the velocity to $0$ to decrease the objective function; this is the optimization analogue of the…
In this paper, we study the gradient descent-ascent method for convex-concave saddle-point problems. We derive a new non-asymptotic global convergence rate in terms of distance to the solution set by using the semidefinite programming…
In this paper, we propose a multi-step inertial Forward--Backward splitting algorithm for minimizing the sum of two non-necessarily convex functions, one of which is proper lower semi-continuous while the other is differentiable with a…
In this paper we propose a variant of the random coordinate descent method for solving linearly constrained convex optimization problems with composite objective functions. If the smooth part of the objective function has Lipschitz…
We consider a family of parallel methods for constrained optimization based on projected gradient descents along individual coordinate directions. In the case of polyhedral feasible sets, local convergence towards a regular solution occurs…
In this paper, we consider continuous-time stochastic optimal control problems where the cost is evaluated through a coherent risk measure. We provide an explicit gradient descent-ascent algorithm which applies to problems subject to…
We study the global convergence of the gradient descent method of the minimization of strictly convex functionals on an open and bounded set of a Hilbert space. Such results are unknown for this type of sets, unlike the case of the entire…
Many classical and modern machine learning algorithms require solving optimization tasks under orthogonality constraints. Solving these tasks with feasible methods requires a gradient descent update followed by a retraction operation on the…
Consider composite nonconvex optimization problems where the objective function consists of a smooth nonconvex term (with Lipschitz-continuous gradient) and a convex (possibly nonsmooth) term. Existing parameter-free methods for such…
We introduce a detailed analysis of the convergence of first-order methods with composite noise (sum of relative and absolute) in gradient for convex and smooth function minimization. This paper illustrates instances of practical problems…
We study the trade-offs between convergence rate and robustness to gradient errors in designing a first-order algorithm. We focus on gradient descent (GD) and accelerated gradient (AG) methods for minimizing strongly convex functions when…
In this letter we study the proximal gradient dynamics. This recently-proposed continuous-time dynamics solves optimization problems whose cost functions are separable into a nonsmooth convex and a smooth component. First, we show that the…
Gradient algorithms are classical in adaptive control and parameter estimation. For instantaneous quadratic cost functions they lead to a linear time-varying dynamic system that converges exponentially under persistence of excitation…
Stochastic coordinate descent algorithms are efficient methods in which each iterate is obtained by fixing most coordinates at their values from the current iteration, and approximately minimizing the objective with respect to the remaining…
Nonconvex optimization is central in solving many machine learning problems, in which block-wise structure is commonly encountered. In this work, we propose cyclic block coordinate methods for nonconvex optimization problems with…
This paper reviews the gradient sampling methodology for solving nonsmooth, nonconvex optimization problems. An intuitively straightforward gradient sampling algorithm is stated and its convergence properties are summarized. Throughout this…
This paper presents a special type of distributed optimization problems, where the summation of agents' local cost functions (i.e., global cost function) is convex, but each individual can be non-convex. Unlike most distributed optimization…