Related papers: Convergence of ADAM for Lipschitz Objective Functi…
We study the convergence issue for the gradient algorithm (employing general step sizes) for optimization problems on general Riemannian manifolds (without curvature constraints). Under the assumption of the local convexity/quasi-convexity…
This paper identifies necessary and sufficient conditions for the exactness of penalty functions in optimization problems whose constraint sets are not necessarily bounded. The case where the data of problems is locally Lipschitz,…
Theoretical estimates of the convergence rate of many well-known gradient-type optimization methods are based on quadratic interpolation, provided that the Lipschitz condition for the gradient is satisfied. In this article we obtain a…
The paper presents a new descent algorithm for locally Lipschitz continuous functions $f:X\to\mathbb{R}$. The selection of a descent direction at some iteration point $x$ combines an approximation of the set-valued gradient of $f$ on a…
This paper deals with two kinds of the one-dimensional global optimization problems over a closed finite interval: (i) the objective function $f(x)$ satisfies the Lipschitz condition with a constant $L$; (ii) the first derivative of $f(x)$…
The paper investigates two inertial extragradient algorithms for seeking a common solution to a variational inequality problem involving a monotone and Lipschitz continuous mapping and a fixed point problem with a demicontractive mapping in…
The generalization performance of deep neural networks with regard to the optimization algorithm is one of the major concerns in machine learning. This performance can be affected by various factors. In this paper, we theoretically prove…
Motivated by applications for simulating quantum many body functions, we propose a new universal ansatz for approximating anti-symmetric functions. The main advantage of this ansatz over previous alternatives is that it is bi-Lipschitz with…
This paper focuses on investigating an inexact stochastic model-based optimization algorithm that integrates preconditioning techniques for solving stochastic composite optimization problems. The proposed framework unifies and extends the…
The paper is devoted to new modifications of recently proposed adaptive methods of Mirror Descent for convex minimization problems in the case of several convex functional constraints. Methods for problems of two classes are considered. The…
Although adaptive optimization algorithms such as Adam show fast convergence in many machine learning tasks, this paper identifies a problem of Adam by analyzing its performance in a simple non-convex synthetic problem, showing that Adam's…
Adam is one of the most influential adaptive stochastic algorithms for training deep neural networks, which has been pointed out to be divergent even in the simple convex setting via a few simple counterexamples. Many attempts, such as…
The paper is devoted to a special Mirror Descent algorithm for problems of convex minimization with functional constraints. The objective function may not satisfy the Lipschitz condition, but it must necessarily have the Lipshitz-continuous…
This paper addresses the study of derivative-free smooth optimization problems, where the gradient information on the objective function is unavailable. Two novel general derivative-free methods are proposed and developed for minimizing…
We propose a new gradient descent algorithm with added stochastic terms for finding the global optimizers of nonconvex optimization problems. A key component in the algorithm is the adaptive tuning of the randomness based on the value of…
Stochastic optimization algorithms using exponential moving averages of the past gradients, such as ADAM, RMSProp and AdaGrad, have been having great successes in many applications, especially in training deep neural networks. ADAM in…
A subgradient method is presented for solving general convex optimization problems, the main requirement being that a strictly-feasible point is known. A feasible sequence of iterates is generated, which converges to within user-specified…
The (global) Lipschitz smoothness condition is crucial in establishing the convergence theory for most optimization methods. Unfortunately, most machine learning and signal processing problems are not Lipschitz smooth. This motivates us to…
Adam is widely adopted in practical applications due to its fast convergence. However, its theoretical analysis is still far from satisfactory. Existing convergence analyses for Adam rely on the bounded smoothness assumption, referred to as…
This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting. A class of stochastic momentum methods, including stochastic gradient descent, heavy ball, and Nesterov's accelerated…