Related papers: Eigenvalue Distribution of Large Random Matrices A…
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We…
The paper deals with distribution of singular values of product of random matrices arising in the analysis of deep neural networks. The matrices resemble the product analogs of the sample covariance matrices, however, an important…
Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important…
We examine the geometry of neural network training using the Jacobian of trained network parameters with respect to their initial values. Our analysis reveals low-dimensional structure in the training process which is dependent on the input…
We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the…
We derive exact analytic expressions for the distributions of eigenvalues and singular values for the product of an arbitrary number of independent rectangular Gaussian random matrices in the limit of large matrix dimensions. We show that…
We consider fully connected and feedforward deep neural networks with dependent and possibly heavy-tailed weights, as introduced in [26], to address limitations of the standard Gaussian prior. It has been proved in [26] that, as the number…
Complex Hermitian random matrices with a unitary symmetry can be distinguished by a weight function. When this is even, it is a known result that the distribution of the singular values can be decomposed as the superposition of two…
Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes…
The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities…
The distribution of the ratios of consecutive eigenvalue spacings of random matrices has emerged as an important tool to study spectral properties of many-body systems. This article numerically investigates the eigenvalue ratios…
We investigate the distribution of eigenvalues of weighted adjacency matrices from a specific ensemble of random graphs. We distribute $N$ vertices across a fixed number $\kappa$ of components, with asymptotically $\alpha_j \dot N$ vertices…
We analyse the limiting behavior of the eigenvalue and singular value distribution for random convolution operators on large (not necessarily Abelian) groups, extending the results by M. Meckes for the Abelian case. We show that for regular…
We study the deformation of the input space by a trained autoencoder via the Jacobians of the trained weight matrices. In doing so, we prove bounds for the mean squared errors for points in the input space, under assumptions regarding the…
Given a collection $\{\lambda_1, \dots, \lambda_n\} $ of real numbers, there is a canonical probability distribution on the set of real symmetric or complex Hermitian matrices with eigenvalues $\lambda_1,\ldots,\lambda_n$. In this paper, we…
Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution,…
Motivated by recent results in random matrix theory we will study the distributions arising from products of complex Gaussian random matrices and truncations of Haar distributed unitary matrices. We introduce an appropriately general class…
For two large matrices ${\mathbf X}$ and ${\mathbf Y}$ with Gaussian i.i.d.\ entries and dimensions $T\times N_X$ and $T\times N_Y$, respectively, we derive the probability distribution of the singular values of $\mathbf{X}^T \mathbf{Y}$ in…
We obtain the asymptotic distribution of eigenvalues of real symmetric tridiagonal matrices as their dimension increases to infinity and whose diagonal and off-diagonal elements asymptotically change with the index n as J_{nt+i nt+i}\sim…
Eigenvalues of Wigner matrices has been a major topic of investigation. A particularly important subclass of such random matrices is formed by the adjacency matrix of an Erd\H{o}s-R\'{e}nyi graph $\mathcal{G}_{n,p}$ equipped with i.i.d.…