Related papers: Convergence Rate Analysis of Accelerated Forward-B…

Optimization plays a key role in machine learning. Recently, stochastic second-order methods have attracted much attention due to their low computational cost in each iteration. However, these algorithms might perform poorly especially if…

While momentum-based optimization algorithms are commonly used in the notoriously non-convex optimization problems of deep learning, their analysis has historically been restricted to the convex and strongly convex setting. In this article,…

A problem of great interest in optimization is to minimize a sum of two closed, proper, and convex functions where one is smooth and the other has a computationally inexpensive proximal operator. In this paper we analyze a family of…

Recently, research on accelerated stochastic gradient descent methods (e.g., SVRG) has made exciting progress (e.g., linear convergence for strongly convex problems). However, the best-known methods (e.g., Katyusha) requires at least two…

We propose NAMA (Newton-type Alternating Minimization Algorithm) for solving structured nonsmooth convex optimization problems where the sum of two functions is to be minimized, one being strongly convex and the other composed with a linear…

We present a forward-backward-based algorithm to minimize a sum of a differentiable function and a nonsmooth function, both being possibly nonconvex. The main contribution of this work is to consider the challenging case where the nonsmooth…

The paper proposes a new algorithm called SymBa that aims to achieve more biologically plausible learning than Back-Propagation (BP). The algorithm is based on the Forward-Forward (FF) algorithm, which is a BP-free method for training…

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide…

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with…

Nesterov's momentum trick is famously known for accelerating gradient descent, and has been proven useful in building fast iterative algorithms. However, in the stochastic setting, counterexamples exist and prevent Nesterov's momentum from…

In this paper we develop algorithms to solve generalized weighted Fermat-Torricelli problems with positive and negative weights and multifacility location problems involving distances generated by Minkowski gauges. We also introduce a new…

Existing methods for nonconvex bilevel optimization (NBO) require prior knowledge of first- and second-order problem-specific parameters (e.g., Lipschitz constants and the Polyak-{\L}ojasiewicz (P{\L}) parameters) to set step sizes, a…

The alternating direction method of multipliers (ADMM) is a popular method for solving convex separable minimization problems with linear equality constraints. The generalization of the two-block ADMM to the three-block ADMM is not trivial…

We consider stochastic optimization problems with non-convex functional constraints, such as those arising in trajectory generation, sparse approximation, and robust classification. To this end, we put forth a recursive momentum-based…

When equipped with efficient optimization algorithms, the over-parameterized neural networks have demonstrated high level of performance even though the loss function is non-convex and non-smooth. While many works have been focusing on…

In recent years, even though Stochastic Gradient Descent (SGD) and its variants are well-known for training neural networks, it suffers from limitations such as the lack of theoretical guarantees, vanishing gradients, and excessive…

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior…

This paper investigates the point convergence of accelerated gradient methods for multiobjective optimization, in both continuous and discrete settings. We address the open problems of whether the solution trajectory of the multiobjective…

This paper describes algorithms for nonnegative matrix factorization (NMF) with the beta-divergence (beta-NMF). The beta-divergence is a family of cost functions parametrized by a single shape parameter beta that takes the Euclidean…

We consider a variable metric and inexact version of the FISTA-type algorithm considered in (Chambolle, Pock, 2016, Calatroni, Chambolle, 2019) for the minimization of the sum of two (possibly strongly) convex functions. The proposed…