Related papers: Neural Collapse with Cross-Entropy Loss

Neural collapse describes the geometry of activation in the final layer of a deep neural network when it is trained beyond performance plateaus. Open questions include whether neural collapse leads to better generalization and, if so, why…

In this paper, we consider the embedding of a complete $d$-uniform geometric hypergraph with $n$ vertices in general position in $\mathbb{R}^d$, where each hyperedge is represented as a $(d-1)$-simplex, and a pair of hyperedges is defined…

We prove rigidity for hypersurfaces with boundary in the unit $(n+1)$-sphere with scalar curvature bounded below by $n(n-1)$. Under appropriate boundary conditions, the hypersurfaces are shown to be part of the equatorial spheres. The lower…

This work presents neural network based minimal entropy closures for the moment system of the Boltzmann equation, that preserve the inherent structure of the system of partial differential equations, such as entropy dissipation and…

Deep learning methods minimise the empirical risk using loss functions such as the cross entropy loss. When minimising the empirical risk, the generalisation of the learnt function still depends on the performance on the training data, the…

We consider perturbation defects obtained by perturbing a 2D conformal field theory (CFT) by a relevant operator on a half-plane. If the perturbed bulk theory flows to an infrared fixed point described by another CFT, the defect flows to a…

Learning generalizable self-supervised graph representations for downstream tasks is challenging. To this end, Contrastive Learning (CL) has emerged as a leading approach. The embeddings of CL are arranged on a hypersphere where similarity…

In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentations. A potential issue of this idea is the existence of…

The recently discovered Neural collapse (NC) phenomenon states that the last-layer weights of Deep Neural Networks (DNN), converge to the so-called Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training. This ETF…

Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building…

A foundational assumption in complex-system collapse studies is that critical transitions are second-order, preceded by early-warning signals like rising autocorrelation, variance, and critical slowing down (Scheffer, 2009). We show this…

In computer vision, it is often observed that formulating regression problems as a classification task often yields better performance. We investigate this curious phenomenon and provide a derivation to show that classification, with the…

Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically…

A formidable perspective in understanding quantum criticality of a given many-body system is through its entanglement contents. Until now, most progress are only limited to the disorder-free case. Here, we develop an efficient scheme to…

In this paper, we introduce matrix entropy as an analytical tool for studying supervised learning, investigating the information content of data representations and classification head vectors, as well as the dynamic interactions between…

Inspired by the boolean discrepancy problem, we study the following optimization problem which we term \textsc{Spherical Discrepancy}: given $m$ unit vectors $v_1, \dots, v_m$, find another unit vector $x$ that minimizes $\max_i \langle x,…

Our study reveals new theoretical insights into over-smoothing and feature over-correlation in graph neural networks. Specifically, we demonstrate that with increased depth, node representations become dominated by a low-dimensional…

This paper demonstrates that in classification problems, fully connected neural networks (FCNs) and residual neural networks (ResNets) cannot be approximated by kernel logistic regression based on the Neural Tangent Kernel (NTK) under…

We study holographic entanglement entropy in four-dimensional quantum gravity with negative cosmological constant. By using the replica trick and evaluating path integrals in the minisuperspace approximation, in conjunction with the…

Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is…