相关论文: Neural Collapse with Cross-Entropy Loss

The fundamental inequality of Guivarc'h relates the entropy and the drift of random walks on groups. It is strict if and only if the random walk does not behave like the uniform measure on balls. We prove that, in any nonelementary…

We propose a neural-network construction of Euclidean scalar quantum field theories from transformer attention heads, defining $n$-point correlators by averaging over random network parameters in the NN-QFT framework. For a single attention…

A novel principle is presented which allows for the proof of bounded weak solutions to a class of physically relevant, strongly coupled parabolic systems exhibiting a formal gradient-flow structure. The main feature of these systems is that…

In the previous papers in this series, the global regularity conjecture for wave maps from two-dimensional Minkowski space $\R^{1+2}$ to hyperbolic space $\H^m$ was reduced to the problem of constructing a minimal-energy blowup solution…

We discuss a possibility that the entire universe on its most fundamental level is a neural network. We identify two different types of dynamical degrees of freedom: "trainable" variables (e.g. bias vector or weight matrix) and "hidden"…

Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are…

We study vectors chosen at random from a compact convex polytope in $\mathbb{R}^n$ given by a finite number of linear constraints. We determine which projections of these random vectors are asymptotically normal as $n\to\infty$. Marginal…

We utilize a recent formulation of a spherically symmetric spacetime endowed with a general decomposition of the energy momentum tensor [Phys. Rev. D, 75, 024031 (2007)] to derive equations governing spherically symmetric distributions of…

Consider a random hypergraph on a set of N vertices in which, for k between 1 and N, a Poisson(N beta_k) number of hyperedges is scattered randomly over all subsets of size k. We collapse the hypergraph by running the following algorithm to…

We derive necessary and sufficient conditions for epi-convergence in distribution of normal integrands. As a basic tool for the proof a new characterisation for distributional convergence of random closed sets is used. Our approach via the…

In this paper we discuss and prove $\epsilon$-regularity theorems for Einstein manifolds $(M^n,g)$, and more generally manifolds with just bounded Ricci curvature, in the collapsed setting. A key tool in the regularity theory of…

We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks. Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of…

We study minimum vertex cover problems on random \alpha-uniform hypergraphs using two different approaches, a replica method in statistical mechanics of random systems and a leaf removal algorithm. It is found that there exists a phase…

We consider recently introduced solutions of Einstein gravity with minimally coupled massless scalars. The geometry is homogeneous, isotropic and asymptotically anti de-Sitter while the scalar fields have linear spatial-dependent profiles.…

Prototype networks provide an intrinsic case based explanation mechanism, but their interpretability is often undermined by prototype collapse, where multiple prototypes degenerate to highly redundant evidence. We attribute this failure…

The \emph{strong collapse} of a simplicial complex, proposed by Barmak and Minian (\emph{Disc. Comp. Geom. 2012}), is a combinatorial collapse of a complex onto its sub-complex. Recently, it has received attention from computational…

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this…

We compute the critical density of collapse for spherically symmetric overdensities in a class of f(R) modified gravity models. For the first time we evolve the Einstein, scalar field and non-linear fluid equations, making the minimal…

Motivated by the potential similarities between the underlying mechanisms of the enhanced memory storage capacity in black holes and in brain networks, we construct an artificial quantum neural network based on gravity-like synaptic…

An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer,…