Related papers: Small Singular Values Matter: A Random Matrix Anal…
We propose a theory of eigenvalues, eigenvectors, singular values, and singular vectors for tensors based on a constrained variational approach much like the Rayleigh quotient for symmetric matrix eigenvalues. These notions are particularly…
Understanding the inner workings of Large Language Models (LLMs) is a critical research frontier. Prior research has shown that a single LLM's concept representations can be captured as steering vectors (SVs), enabling the control of LLM…
The Restricted Boltzmann Machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we…
Multivariate statistical analysis is concerned with observations on several variables which are thought to possess some degree of inter-dependence. Driven by problems in genetics and the social sciences, it first flowered in the earlier…
Large Language Models (LLMs) are pretrained on massive datasets and later instruction-tuned via supervised fine-tuning (SFT) or reinforcement learning (RL). Best practices emphasize large, diverse pretraining data, whereas post-training…
Masked Language Modeling (MLM) is widely used to pretrain language models. The standard random masking strategy in MLM causes the pre-trained language models (PLMs) to be biased toward high-frequency tokens. Representation learning of rare…
We consider the singular vectors of any $m \times n$ submatrix of a rectangular $M \times N$ Gaussian matrix and study their asymptotic overlaps with those of the full matrix, in the macroscopic regime where $N \,/\, M\,$, $m \,/\, M$ as…
Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple…
The inference of a large symmetric signal-matrix $\mathbf{S} \in \mathbb{R}^{N\times N}$ corrupted by additive Gaussian noise, is considered for two regimes of growth of the rank $M$ as a function of $N$. For sub-linear ranks…
We take on a Random Matrix theory viewpoint to study the spectrum of certain reversible Markov chains in random environment. As the number of states tends to infinity, we consider the global behavior of the spectrum, and the local behavior…
This article carries out a large dimensional analysis of standard regularized discriminant analysis classifiers designed on the assumption that data arise from a Gaussian mixture model with different means and covariances. The analysis…
Random-matrix theory is applied to transition-rate matrices in the Pauli master equation. We study the distribution and correlations of eigenvalues, which govern the dynamics of complex stochastic systems. Both the cases of identical and of…
Spectral properties of random matrices play an important role in statistics, machine learning, communications, and many other areas. Engaging results regarding the convergence of the empirical spectral distribution (ESD) and the…
This paper develops a unified analytical framework for determinant identities under finite-rank perturbations of square matrices that remains valid without invertibility assumptions. In contrast to classical inverse-based formulations, the…
In the recent paper \cite{1}, Denton et al. provided the eigenvector-eigenvalue identity for Hermitian matrices, and a survey was also given for such identity in the literature. The main aim of this paper is to present the identity related…
The spectral statistics of non-Hermitian random matrices are of importance as a diagnostic tool for chaotic behavior in open quantum systems. Here, we investigate the statistical properties of singular values in non-Hermitian random…
Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which prevents their practical deployment in many applications.…
Truncated singular value decomposition is a reduced version of the singular value decomposition in which only a few largest singular values are retained. This paper presents a novel perturbation analysis for the truncated singular value…
Metric regularity is among the central concepts of nonlinear and variational analysis, constrained optimization, and their numerous applications. However, metric regularity can be elusive for some important ill-posed classes of problems…
Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where…