Related papers: The conjugate gradient algorithm on a general clas…

In this paper, we investigate a general class of stochastic gradient descent (SGD) algorithms, called Conditioned SGD, based on a preconditioning of the gradient direction. Using a discrete-time approach with martingale tools, we establish…

We consider the Laguerre Unitary Ensemble (aka, Wishart Ensemble) of sample covariance matrices $A = XX^*$, where $X$ is an $N \times n$ matrix with iid standard complex normal entries. Under the scaling $n = N + \lfloor \sqrt{ 4 c N}…

We prove novel convergence results for a stochastic proximal gradient algorithm suitable for solving a large class of convex optimization problems, where a convex objective function is given by the sum of a smooth and a possibly non-smooth…

Stochastic gradient algorithm is a key ingredient of many machine learning methods, particularly appropriate for large-scale learning.However, a major caveat of large data is their incompleteness.We propose an averaged stochastic gradient…

This article concerns a class of generalized linear mixed models for clustered data, where the random effects are mapped uniquely onto the grouping structure and are independent between groups. We derive necessary and sufficient conditions…

We study the convergence of a variant of distributed gradient descent (DGD) on a distributed low-rank matrix approximation problem wherein some optimization variables are used for consensus (as in classical DGD) and some optimization…

Suppose $X_p$ is a real $p \times n$ matrix with independent entries and consider the (unscaled) sample covariance matrix $S_p=X_pX_p^T$. The Marchenko-Pastur law was discovered as the limit of the bulk distribution of the sample covariance…

In this article, we rigorously establish the consistency of generalized cross-validation as a parameter-choice rule for solving inverse problems. We prove that the index chosen by leave-one-out GCV achieves a non-asymptotic, order-optimal…

Motivated by broad applications in reinforcement learning and machine learning, this paper considers the popular stochastic gradient descent (SGD) when the gradients of the underlying objective function are sampled from Markov processes.…

We give a derivation of the method of conjugate gradients based on the requirement that each iterate minimizes a strictly convex quadratic on the space spanned by the previously observed gradients. Rather than verifying that the search…

In this article, we propose an algorithm for the nonlinear conjugate gradient method to find a Pareto critical point of unconstrained multiobjective interval optimization problems. In this algorithm, we use the Wolfe line search procedure…

In this paper, we propose a unified convergence analysis for a class of generic shuffling-type gradient methods for solving finite-sum optimization problems. Our analysis works with any sampling without replacement strategy and covers many…

We introduce a family of coefficients based on U-statistics that generalize the notion of correlation and explore their properties in the large dimensional multivariate case, showing that in the null case of uncorrelated variables, the…

Deflation techniques for Krylov subspace methods have seen a lot of attention in recent years. They provide means to improve the convergence speed of these methods by enriching the Krylov subspace with a deflation subspace. The most common…

We obtain a recurrence relation in $d$ for the average singular value $% \alpha (d)$ of a complex valued $d\times d$\ matrix $\frac{1}{\sqrt{d}}X$ with random i.i.d., N( 0,1) entries, and use it to show that $\alpha (d)$ decreases…

In this contribution, we present a full overview of the continuous stochastic gradient (CSG) method, including convergence results, step size rules and algorithmic insights. We consider optimization problems in which the objective function…

In this paper, we derive a joint central limit theorem for random vector whose components are function of random sesquilinear forms. This result is a natural extension of the existing central limit theory on random quadratic forms. We also…

Spectral correlations in unitary invariant, non-Gaussian ensembles of large random matrices possessing an eigenvalue gap are studied within the framework of the orthogonal polynomial technique. Both local and global characteristics of…

We study problem-dependent rates, i.e., generalization errors that scale near-optimally with the variance, the effective loss, or the gradient norms evaluated at the "best hypothesis." We introduce a principled framework dubbed "uniform…

In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms. Specifically, there are $K$ clients whose individually chosen models are…