Related papers: Can Single-Shuffle SGD be Better than Reshuffling …

We analyze the convergence of the Conjugate Gradient (CG) method in exact arithmetic, when the coefficient matrix $A$ is symmetric positive semidefinite and the system is consistent. To do so, we diagonalize $A$ and decompose the algorithm…

In recent years, there has been considerable interest in designing stochastic first-order algorithms to tackle finite-sum smooth minimax problems. To obtain the gradient estimates, one typically relies on the uniform…

Variational inequalities have gained significant attention in machine learning and optimization research. While stochastic methods for solving these problems typically assume independent data sampling, we investigate an alternative approach…

We study the spectral properties of a class of random matrices of the form $S_n^{-} = n^{-1}(X_1 X_2^* - X_2 X_1^*)$ where $X_k = \Sigma_k^{1/2}Z_k$, $Z_k$'s are independent $p\times n$ complex-valued random matrices, and $\Sigma_k$ are…

In this paper, we investigate a general class of stochastic gradient descent (SGD) algorithms, called Conditioned SGD, based on a preconditioning of the gradient direction. Using a discrete-time approach with martingale tools, we establish…

Mechanical shufflers used in many casinos employ a card shuffling scheme called \emph{shelf shuffling}. In a single-shelf shuffling, cards arrive sequentially, and each incoming card is independently placed on the top or the bottom of a…

Recht and R\'{e} introduced the noncommutative arithmetic geometric mean inequality (NC-AGM) for matrices with a constant depending on the degree $d$ and the dimension $m$. In this paper we prove AGM inequalities with a dimension-free…

Random matrices like GUE, GOE and GSE have been studied for decades and have been shown that they possess a lot of nice properties. In 2005, a new property of independent GUE random matrices is discovered by Haagerup and Thorbj{\o}rnsen in…

We establish sharp stability results for of non--selfadjoint the ascent and descent spectra under strong resolvent convergence (SRS), a natural framework for finite element approximations of non-selfadjoint and singularly perturbed…

An equation is obtained for the Stieltjes transform of the normalized distribution of singular values of non-symmetric band random matrices in the limit when the band width and rank of the matrix simultaneously tend to infinity. Conditions…

Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving the empirical performance gap insufficiently explained. In…

The Singular Value Decomposition is a matrix decomposition technique widely used in the analysis of multivariate data, such as complex space-time images obtained in both physical and biological systems. In this paper, we examine the…

We study the asymptotic behavior of the spectra of matrices of the form $S_n = \frac{1}{n}XX^*$ where $X =\sum_{r=1}^K X_r$, where $X_r = A_r^\frac{1}{2}Z_rB_r^\frac{1}{2}$, $K \in \mathbb{N}$ and $A_r,B_r$ are sequences of positive…

This paper aims to develop a simple procedure to reduce and control the condition number of random matrices, and investigate the effect on the persistent homology (PH) of point clouds of well- and ill-conditioned matrices. For a square…

Anomalous diffusion occurs in many physical and biological phenomena, when the growth of the mean squared displacement (MSD) with time has an exponent different from one. We show that recurrent neural networks (RNN) can efficiently…

Let $G$ be a multiplicative subsemigroup of the general linear group $\Gl(\mathbb{R}^d)$ which consists of matrices with positive entries such that every column and every row contains a strictly positive element. Given a $G$--valued random…

Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in runtime as it waits for the slowest workers (stragglers). Asynchronous methods can alleviate stragglers, but cause gradient staleness…

We analyze Local SGD (aka parallel or federated SGD) and Minibatch SGD in the heterogeneous distributed setting, where each machine has access to stochastic gradient estimates for a different, machine-specific, convex objective; the goal is…

We consider whether algorithmic choices in over-parameterized linear matrix factorization introduce implicit regularization. We focus on noiseless matrix sensing over rank-$r$ positive semi-definite (PSD) matrices in $\mathbb{R}^{n \times…

We show that a perturbation of any fixed square matrix D by a random unitary matrix is well invertible with high probability. A similar result holds for perturbations by random orthogonal matrices; the only notable exception is when D is…