Related papers: Geometric and Spectral Alignment for Deep Neural N…
This paper develops the angular and static-channel component of Geometric and Spectral Alignment for residual Jacobian chains. Starting from Cartan-coordinate rigidity and fitted effective-rank windows, we study how dominant singular…
We consider a family of deep neural networks consisting of two groups of convolutional layers, a downsampling operator, and a fully connected layer. The network structure depends on two structural parameters which determine the numbers of…
The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities…
Nirenberg's famous complex Frobenius theorem gives necessary and sufficient conditions on a locally integrable structure for when the manifold is locally diffeomorphic to $\mathbb R^r\times\mathbb C^m\times \mathbb R^{N-r-2m}$ through a…
Our goal is to convince the readers that the theory of complex normal surface singularities can be a powerful tool in the study of numerical semigroups, and, in the same time, a very rich source of interesting affine and numerical…
We show that given an estimate $\widehat{A}$ that is close to a general high-rank positive semi-definite (PSD) matrix $A$ in spectral norm (i.e., $\|\widehat{A}-A\|_2 \leq \delta$), the simple truncated SVD of $\widehat{A}$ produces a…
Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes…
As part of his celebrated Complex Frobenius Theorem, Nirenberg showed that given a smooth elliptic structure (on a smooth manifold), the manifold is locally diffeomorphic to an open subset of $\mathbb{R}^r\times \mathbb{C}^n$ (for some $r$…
Graph representation learning has become a standard approach for analyzing networked data, with latent embeddings widely used for link prediction, community detection, and related tasks. Yet a basic design choice, the latent dimension, is…
Understanding why gradient-based training in deep networks exhibits strong implicit bias remains challenging, in part because tractable singular-value dynamics are typically available only for balanced deep linear models. We propose an…
We propose a novel low-rank initialization framework for training low-rank deep neural networks -- networks where the weight parameters are re-parameterized by products of two low-rank matrices. The most successful prior existing approach,…
This work is motivated by the relation between the KP and BKP integrable hierarchies, whose $\tau$-functions may be viewed as sections of dual determinantal and Pfaffian line bundles over infinite dimensional Grassmannians. In finite…
Deep neural networks are notorious for defying theoretical treatment. However, when the number of parameters in each layer tends to infinity, the network function is a Gaussian process (GP) and quantitatively predictive description is…
The Hessian of a neural network captures parameter interactions through second-order derivatives of the loss. It is a fundamental object of study, closely tied to various problems in deep learning, including model design, optimization, and…
Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that for feed forward neural networks under the cross entropy…
This article considers fully connected neural networks with Gaussian random weights and biases as well as $L$ hidden layers, each of width proportional to a large parameter $n$. For polynomially bounded non-linearities we give sharp…
A Cartan-geometric, jet bundle formulation of curvature-aware variance bounds in parametric statistical estimation is developed. Building on our earlier extrinsic Hilbert space approach to the Cram\'er-Rao and Bhattacharyya-type…
Modern deep networks are heavily overparameterized yet often generalize well, suggesting a form of low intrinsic complexity not reflected by parameter counts. We study this complexity at initialization through the effective rank of the…
We consider convex maps f:R^n -> R^n that are monotone (i.e., that preserve the product ordering of R^n), and nonexpansive for the sup-norm. This includes convex monotone maps that are additively homogeneous (i.e., that commute with the…
This work analyzes Graph Neural Networks, a generalization of Fully-Connected Deep Neural Nets on Graph structured data, when their width, that is the number of nodes in each fullyconnected layer is increasing to infinity. Infinite Width…