Related papers: Inverting the Leverage Score Gradient: An Efficien…
This paper argues that the method of least squares has significant unfulfilled potential in modern machine learning, far beyond merely being a tool for fitting linear models. To release its potential, we derive custom gradients that…
Gradient-based dimension reduction decreases the cost of Bayesian inference and probabilistic modeling by identifying maximally informative (and informed) low-dimensional projections of the data and parameters, allowing high-dimensional…
The paper studies the solution of stochastic optimization problems in which approximations to the gradient and Hessian are obtained through subsampling. We first consider Newton-like methods that employ these approximations and discuss how…
Recent development on mixed precision techniques has largely enhanced the performance of various linear algebra solvers, one of which being the solver for the least squares problem $\min_{x}\lVert b-Ax\rVert_{2}$. By transforming least…
Sparse inverse covariance selection is a fundamental problem for analyzing dependencies in high dimensional data. However, such a problem is difficult to solve since it is NP-hard. Existing solutions are primarily based on convex…
We present a novel statistical inference framework for convex empirical risk minimization, using approximate stochastic Newton steps. The proposed algorithm is based on the notion of finite differences and allows the approximation of a…
We extend the geometrical inverse approximation approach for solving linear least-squares problems. For that we focus on the minimization of $1-\cos(X(A^TA),I)$, where $A$ is a given rectangular coefficient matrix and $X$ is the approximate…
Nonlinear least-squares problems are a special class of unconstrained optimization problems in which their gradient and Hessian have special structures. In this paper, we exploit these structures and proposed a matrix-free algorithm with a…
Non-linear least squares solvers are used across a broad range of offline and real-time model fitting problems. Most improvements of the basic Gauss-Newton algorithm tackle convergence guarantees or leverage the sparsity of the underlying…
This work considers the non-convex finite sum minimization problem. There are several algorithms for such problems, but existing methods often work poorly when the problem is badly scaled and/or ill-conditioned, and a primary goal of this…
We consider the rank minimization problem from quadratic measurements, i.e., recovering a rank $r$ matrix $X \in \mathbb{R}^{n \times r}$ from $m$ scalar measurements $y_i=a_i^{\top} XX^{\top} a_i,\;a_i\in \mathbb{R}^n,\;i=1,\ldots,m$. Such…
In this paper we apply the stochastic variance reduced gradient (SVRG) method, which is a popular variance reduction method in optimization for accelerating the stochastic gradient method, to solve large scale linear ill-posed systems in…
We consider the stochastic approximation problem where a convex function has to be minimized, given only the knowledge of unbiased estimates of its gradients at certain points, a framework which includes machine learning methods based on…
We consider estimation in the single index model where the link function is monotone. For this model a profile least squares estimator has been proposed to estimate the unknown link function and index. Although it is natural to propose this…
New recursive least squares algorithms with rank two updates (RLSR2) that include both exponential and instantaneous forgetting (implemented via a proper choice of the forgetting factor and the window size) are introduced and systematically…
In this paper, we consider a statistical problem of learning a linear model from noisy samples. Existing work has focused on approximating the least squares solution by using leverage-based scores as an importance sampling distribution.…
This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest $\mathbf{x}^{\natural}\in\mathbb{R}^{n}$ from $m$ quadratic equations/samples…
We study finite-sum nonconvex optimization problems, where the objective function is an average of $n$ nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with…
In this work we investigate the practicality of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the machine learning literature…
Consider the generalized linear least squares (GLS) problem $\min\|Lx\|_2 \ \mathrm{s.t.} \ \|M(Ax-b)\|_2=\min$. The weighted pseudoinverse $A_{ML}^{\dag}$ is the matrix that maps $b$ to the minimum 2-norm solution of this GLS problem. By…