Related papers: Convergence Properties of Nonlinear GMRES Applied …
We present a generalization of Nesterov's accelerated gradient descent algorithm. Our algorithm (AGNES) provably achieves acceleration for smooth convex and strongly convex minimization tasks with noisy gradient estimates if the noise…
We establish the existence theory of several commonly used finite element (FE) nonlinear fully discrete solutions, and the convergence theory of a linearized iteration. First, it is shown for standard FE, SUPG and edge-averaged method…
Consider solving large sparse range symmetric singular linear systems $ A {\bf x}= {\bf b} $ which arise, for instance, in the discretization of convection diffusion equations with periodic boundary conditions, and partial differential…
Non-maximum suppression (NMS) is used in virtually all state-of-the-art object detection pipelines. While essential object detection ingredients such as features, classifiers, and proposal methods have been extensively researched…
We propose a novel algorithm based on inexact GMRES methods for linear response calculations in density functional theory. Such calculations require iteratively solving a nested linear problem $\mathcal{E} \delta\rho = b$ to obtain the…
This paper investigates the point convergence of accelerated gradient methods for multiobjective optimization, in both continuous and discrete settings. We address the open problems of whether the solution trajectory of the multiobjective…
The Nesterov accelerated gradient method, introduced in 1983, has been a cornerstone of optimization theory and practice. Yet the question of its point convergence had remained open. In this work, we resolve this longstanding open problem…
The Regularized Nonlinear Acceleration (RNA) algorithm is an acceleration method capable of improving the rate of convergence of many optimization schemes such as gradient descend, SAGA or SVRG. Until now, its analysis is limited to convex…
We present variants of the Conjugate Gradient (CG), Conjugate Residual (CR), and Generalized Minimal Residual (GMRES) methods which are both pipelined and flexible. These allow computation of inner products and norms to be overlapped with…
Recent research has indicated a substantial rise in interest in understanding Nesterov's accelerated gradient methods via their continuous-time models. However, most existing studies focus on specific classes of Nesterov's methods, which…
The linear algorithm of the Wiener filter and constrained realizations (CRs) of Gaussian random fields is extended here to perform non-linear CRs. The procedure consists of: (1) Using low resolution data to constrain a high resolution…
By using a time-dependent operator converting a distribution function (statistical operator) of a total system under consideration into the relevant form, new exact nonlinear generalized master equations (GMEs) are derived. The…
Nesterov's accelerated gradient method for minimizing a smooth strongly convex function $f$ is known to reduce $f(\x_k)-f(\x^*)$ by a factor of $\eps\in(0,1)$ after $k\ge O(\sqrt{L/\ell}\log(1/\eps))$ iterations, where $\ell,L$ are the two…
We describe a convergence acceleration technique for unconstrained optimization problems. Our scheme computes estimates of the optimum from a nonlinear average of the iterates produced by any optimization method. The weights in this average…
New convergence bounds are presented for weighted, preconditioned, and deflated GMRES for the solution of large, sparse, non-Hermitian linear systems. These bounds are given for the case when the Hermitian part of the coefficient matrix is…
Since Nesterov's seminal 1983 work, many accelerated first-order optimization methods have been proposed, but their analyses lacks a common unifying structure. In this work, we identify a geometric structure satisfied by a wide range of…
Our work presents a new iterative scheme to approximate the fixed points of nonexpansive mapping. The proposed algorithm is constructed to enhance convergence efficiency while preserving theoretical robustness. Under appropriate assumptions…
Neumann series underlie both Krylov methods and algebraic multigrid smoothers. A low-synch modified Gram-Schmidt (MGS)-GMRES algorithm is described that employs a Neumann series to accelerate the projection step. A corollary to the backward…
It is well known that as a famous type of iterative methods in numerical linear algebra, Gauss-Seidel iterative methods are convergent for linear systems with strictly or irreducibly diagonally dominant matrices, invertible $H-$matrices…
In this paper, we study the stochastic gradient descent (SGD) method for the nonconvex nonsmooth optimization, and propose an accelerated SGD method by combining the variance reduction technique with Nesterov's extrapolation technique.…