Related papers: Small Singular Values Matter: A Random Matrix Anal…
A low rank matrix X has been contaminated by uniformly distributed noise, missing values, outliers and corrupt entries. Reconstruction of X from the singular values and singular vectors of the contaminated matrix Y is a key problem in…
We study largest singular values of large random matrices, each with mean of a fixed rank $K$. Our main result is a limit theorem as the number of rows and columns approach infinity, while their ratio approaches a positive constant. It…
We show that singular value decomposition of the lm_head} weight matrix of a transformer-based large language model -- requiring only five lines of PyTorch and no model inference -- reveals interpretable semantic subspaces directly from the…
Large-scale foundation models have demonstrated remarkable versatility across a wide range of downstream tasks. However, fully fine-tuning these models incurs prohibitive computational costs, motivating the development of…
The paper is concerned with deformed Wigner random matrices. These matrices are closely related to Deep Neural Networks (DNNs): weight matrices of trained DNNs could be represented in the form $R + S$, where $R$ is random and $S$ is highly…
Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model. LLMs contain billions of parameters, so these small fractions, such…
Complex Hermitian random matrices with a unitary symmetry can be distinguished by a weight function. When this is even, it is a known result that the distribution of the singular values can be decomposed as the superposition of two…
We propose new algorithms for singular value decomposition (SVD) of very large-scale matrices based on a low-rank tensor approximation technique called the tensor train (TT) format. The proposed algorithms can compute several dominant…
In the probabilistic topic models, the quantity of interest---a low-rank matrix consisting of topic vectors---is hidden in the text corpus matrix, masked by noise, and the Singular Value Decomposition (SVD) is a potentially useful tool for…
Let $(\varepsilon_{t})_{t>0}$ be a sequence of independent real random vectors of $p$-dimension and let $X_T=\sum_{t=s+1}^{s+T}\varepsilon_t\varepsilon^T_{t-s}/T$ be the lag-$s$ ($s$ is a fixed positive integer) auto-covariance matrix of…
Many complex systems can be reduced to their key components through spectrally decomposing matrices that capture their dynamics. These matrices can in turn be constructed from data, often by least-squares fitting: examples of algorithms to…
Post-training fundamentally alters the behavior of large language models (LLMs), yet its impact on the internal parameter space remains poorly understood. In this work, we conduct a systematic singular value decomposition (SVD) analysis of…
An important task when processing sensor data is to distinguish relevant from irrelevant data. This paper describes a method for an iterative singular value decomposition that maintains a model of the background via singular vectors…
Let $\a$ be a real-valued random variable of mean zero and variance 1. Let $M_n(\a)$ denote the $n \times n$ random matrix whose entries are iid copies of $\a$ and $\sigma_n(M_n(\a))$ denote the least singular value of $M_n(\a)$.…
A recent development in random matrix theory, the intrinsic freeness principle, establishes that the spectrum of very general random matrices behaves as that of an associated free operator. This reduces the study of such random matrices to…
Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5…
The measurement data collected from the supervisory control and data acquisition (SCADA) system installed in distribution network can reflect the operational state of the network effectively. In this paper, a random matrix theory (RMT)…
Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as…
Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data and rely on overparameterized models, where classical low-dimensional intuitions break down. In particular, the proportional regime where…
Random Matrix Theory (RMT) has successfully modeled diverse systems, from energy levels of heavy nuclei to zeros of $L$-functions. Many statistics in one can be interpreted in terms of quantities of the other; for example, zeros of…