Related papers: Can Single-Shuffle SGD be Better than Reshuffling …

We develop new techniques for proving lower bounds on the least singular value of random matrices with limited randomness. The matrices we consider have entries that are given by polynomials of a few underlying base random variables. This…

We study an example of a {\em hit-and-run} random walk on the symmetric group $\mathbf S_n$. Our starting point is the well understood {\em top-to-random} shuffle. In the hit-and-run version, at each {\em single step}, after picking the…

Inspired by a common technique for shuffling a deck of cards on a table without riffling, we formalize the pile shuffle and investigate its capabilities as a sorting device. Our study is novel in that we consider pile shuffle in three…

We consider the single eigenvalue fluctuations of random matrices of general Wigner-type, under a one-cut assumption on the density of states. For eigenvalues in the bulk, we prove that the asymptotic fluctuations of a single eigenvalue…

Understanding the algorithmic bias of \emph{stochastic gradient descent} (SGD) is one of the key challenges in modern machine learning and deep learning theory. Most of the existing works, however, focus on \emph{very small or even…

Parallel-across-the method time integration can provide small scale parallelism when solving initial value problems. Spectral deferred corrections (SDC) with a diagonal sweeper, which is closely related to iterated Runge-Kutta methods…

The Randomized Singular Value Decomposition (RSVD) is a widely used algorithm for efficiently computing low-rank approximations of large matrices, without the need to construct a full-blown SVD. Of interest, of course, is the approximation…

We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show that the rate of convergence in all cases consists of two…

Stochastic Primal-Dual Hybrid Gradient (SPDHG) is an algorithm proposed by Chambolle et al. (2018) to efficiently solve a wide class of nonsmooth large-scale optimization problems. In this paper we contribute to its theoretical foundations…

Auslander-Reiten conjecture, which says that an Artin algebra does not have any non-projective generator with vanishing self-extensions in all positive degrees, is shown to be invariant under certain singular equivalences induced by adjoint…

Machine learning models trained with \emph{stochastic} gradient descent (SGD) can generalize better than those trained with deterministic gradient descent (GD). In this work, we study SGD's impact on generalization through the lens of the…

The Schatten-$p$ norm ($0<p<1$) has been widely used to replace the nuclear norm for better approximating the rank function. However, existing methods are either 1) not scalable for large scale problems due to relying on singular value…

We present some extensions of Bernstein's concentration inequality for random matrices. This inequality has become a useful and powerful tool for many problems in statistics, signal processing and theoretical computer science. The main…

We revisit the sequential rate-distortion (SRD) trade-off problem for vector-valued Gauss-Markov sources with mean-squared error distortion constraints. We show via a counterexample that the dynamic reverse water-filling algorithm suggested…

This study investigates rare event detection on tabular data within binary classification. Standard techniques to handle class imbalance include SMOTE, which generates synthetic samples from the minority class. However, SMOTE is…

We consider alternating gradient descent (AGD) with fixed step size applied to the asymmetric matrix factorization objective. We show that, for a rank-$r$ matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$, $T = C…

Let $M_n$ be a class of symmetric sparse random matrices, with independent entries $M_{ij} = \delta_{ij} \xi_{ij}$ for $i \leq j$. $\delta_{ij}$ are i.i.d. Bernoulli random variables taking the value $1$ with probability $p \geq…

Except the Toeplitz and Hankel matrices, the common patterned matrices for which the limiting spectral distribution (LSD) are known to exist, share a common property--the number of times each random variable appears in the matrix is (more…

We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classical SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point…

SGD with Momentum (SGDM) is a widely used family of algorithms for large-scale optimization of machine learning problems. Yet, when optimizing generic convex functions, no advantage is known for any SGDM algorithm over plain SGD. Moreover,…