Related papers: Limited-memory BFGS Systems with Diagonal Updates

This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS where stochastic batches…

Matrix Factorization is a popular non-convex optimization problem, for which alternating minimization schemes are mostly used. They usually suffer from the major drawback that the solution is biased towards one of the optimization…

We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning. Contrary to the classical variants of these methods that sequentially build…

This paper considers the regularization continuation method and the trust-region updating strategy for the optimization problem with linear equality constraints.The proposed method utilizes the linear conservation law of the regularization…

The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each…

A MATLAB implementation of the More-Sorensen sequential (MSS) method is presented. The MSS method computes the minimizer of a quadratic function defined by a limited-memory BFGS matrix subject to a two-norm trust-region constraint. This…

Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O$((1/\sqrt{t})^t)$. The methods that obtain this rate, however, exhibit a well-known…

We consider systems of linear differential and difference equations \begin{eqnarray*} \partial Y(x) =A(x)Y(x), \sigma Y(x) =B(x)Y(x) \end{eqnarray*} with $\partial = \frac{d}{dx}$, $\sigma$ a shift operator $\sigma(x) = x+a$, $q$-dilation…

Reinforcement Learning (RL) algorithms allow artificial agents to improve their action selections so as to increase rewarding experiences in their environments. Deep Reinforcement Learning algorithms require solving a nonconvex and…

We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a…

Lagrangian duality in mixed integer optimization is a useful framework for problems decomposition and for producing tight lower bounds to the optimal objective, but in contrast to the convex counterpart, it is generally unable to produce…

In this paper we proposed quasi-Newton and limited memory quasi-Newton methods for objective functions defined on Grassmannians or a product of Grassmannians. Specifically we defined BFGS and L-BFGS updates in local and global coordinates…

Linear systems in applications are typically well-posed, and yet the coefficient matrices may be nearly singular in that the condition number $\kappa(\boldsymbol{A})$ may be close to $1/\varepsilon_{w}$, where $\varepsilon_{w}$ denotes the…

The existing low-memory BLS implementation proposed recently avoids the need for storing and inverting large matrices, to achieve efficient usage of memories. However, the existing low-memory BLS implementation sacrifices the testing…

In this paper, we study the complexity of solving generic over-determined bilinear systems over a finite field $\mathbb{F}$. Given a generic bilinear sequence $B \in \mathbb{F}[\mathbf{x},\mathbf{y}]$, with respect to a partition of…

We propose a novel limited-memory stochastic block BFGS update for incorporating enriched curvature information in stochastic approximation methods. In our method, the estimate of the inverse Hessian matrix that is maintained by it, is…

The efficient solution of large-scale multiterm linear matrix equations is a challenging task in numerical linear algebra, and it is a largely open problem. We propose a new iterative scheme for symmetric and positive definite operators,…

The motivation to study the behavior of limited-memory BFGS (L-BFGS) on nonsmooth optimization problems is based on two empirical observations: the widespread success of L-BFGS in solving large-scale smooth optimization problems, and the…

Inspired by Karr's algorithm, we consider the summations involving a sequence satisfying a recurrence of order two. The structure of such summations provides an algebraic framework for solving the difference equations of form…

We consider regularized cutting-plane methods to minimize a convex function that is the sum of a large number of component functions. One important example is the dual problem obtained from Lagrangian relaxation on a decomposable problem.…