Related papers: Anderson acceleration with adaptive relaxation for…
Distributed optimization, where the computations are performed in a localized and coordinated manner using multiple agents, is a promising approach for solving large-scale optimization problems, e.g., those arising in model predictive…
Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems. Up to a primal-dual transformation, it is also the same as accelerated stochastic gradient descent…
This paper deals with speeding up the convergence of a class of two-step iterative methods for solving linear systems of equations. To implement the acceleration technique, the residual norm associated with computed approximations for each…
Optimization plays a key role in machine learning. Recently, stochastic second-order methods have attracted much attention due to their low computational cost in each iteration. However, these algorithms might perform poorly especially if…
This article is devoted to one particular case of using universal accelerated proximal envelopes to obtain computationally efficient accelerated versions of methods used to solve various optimization problem setups. In this paper, we…
In this paper, we study a nonconvex continuous relaxation of MAP inference in discrete Markov random fields (MRFs). We show that for arbitrary MRFs, this relaxation is tight, and a discrete stationary point of it can be easily reached by a…
Following the seminal work of Nesterov, accelerated optimization methods have been used to powerfully boost the performance of first-order, gradient-based parameter estimation in scenarios where second-order optimization strategies are…
We propose two new alternating direction methods to solve "fully" nonsmooth constrained convex problems. Our algorithms have the best known worst-case iteration-complexity guarantee under mild assumptions for both the objective residual and…
In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic gradient descent for…
In this paper we accomplish the development of the fast rank-adaptive solver for tensor-structured symmetric positive definite linear systems in higher dimensions. In [arXiv:1301.6068] this problem is approached by alternating minimization…
Nonlinear convex problems arise in various areas of applied mathematics and engineering. Classical techniques such as the relaxed proximal point algorithm (PPA) and the prediction correction (PC) method were proposed for linearly…
This article is devoted to one particular case of using universal accelerated proximal envelopes to obtain computationally efficient accelerated versions of methods used to solve various optimization problem setups. We propose a proximally…
The recently proposed Activation Relaxation (AR) algorithm provides a simple and robust approach for approximating the backpropagation of error algorithm using only local learning rules. Unlike competing schemes, it converges to the exact…
We propose a new simple convergence acceleration method for wide range class of convergent alternating series. It has some common features with Smith's and Ford's modification of Levin's and Weniger's sequence transformations, but its…
This paper considers the numerical solution of generalized Sylvester matrix equations, which arise in many scientific and engineering applications but remain challenging to solve efficiently, particularly when the coefficient matrices are…
In this paper, we provide a rigorous proof of convergence of the Adaptive Moment Estimate (Adam) algorithm for a wide class of optimization objectives. Despite the popularity and efficiency of the Adam algorithm in training deep neural…
This paper primarily presents numerical results for the Anderson accelerated Newton method on a set of benchmark problems. The results demonstrate superlinear convergence to solutions of both degenerate and nondegenerate problems. The…
In this paper, we propose a unified two-phase scheme to accelerate any high-order regularized tensor approximation approach on the smooth part of a composite convex optimization model. The proposed scheme has the advantage of not needing to…
The natural exponential function is widely used in modeling many engineering and scientific systems. It is also an integral part of many neural network activation function such as sigmoid, tanh, ELU, RBF etc. Dedicated hardware accelerator…
The Newton's method for solving stationary Navier-Stokes equations (NSE) is known to convergent fast, however, may fail due to a bad initial guess. This work presents a simple-to-implement nonlinear preconditioning of Newton's iteration,…