Related papers: Neural Collapse with Cross-Entropy Loss

Even though Deep Neural Networks (DNNs) are widely celebrated for their practical performance, they possess many intriguing properties related to depth that are difficult to explain both theoretically and intuitively. Understanding how…

The identification of a causal-connection scale motivates us to propose a new covariant bound on entropy within a generic space-like region. This "causal entropy bound", scaling as the square root of EV, and thus lying around the geometric…

Although deep neural networks achieve tremendous success on various classification tasks, the generalization ability drops sheer when training datasets exhibit long-tailed distributions. One of the reasons is that the learned…

Neural image classification models typically consist of two components. The first is an image encoder, which is responsible for encoding a given raw image into a representative vector. The second is the classification component, which is…

Over 15 million epilepsy patients worldwide do not respond to drugs and require surgical treatment. Successful surgical treatment requires complete removal, or disconnection of the epileptogenic zone (EZ), but without a prospective…

We consider first passage percolation on sparse random graphs with prescribed degree distributions and general independent and identically distributed edge weights assumed to have a density. Assuming that the degree distribution satisfies a…

Neural collapse (NC) describes the structured geometry that emerges in the features and weights of trained classifiers. Recent theory suggests NC can be suboptimal in deep architectures, attributing this to an explicit low-rank bias from L2…

In [5], Colding-Ilmanen-Minicozzi-White showed that within the class of closed smooth self-shrinkers in $\mathbb{R}^{n+1}$, the entropy is uniquely minimized at the round sphere. They conjectured that, for $2\leq n\leq 6$, the round sphere…

An open question in systems and computational neuroscience is how neural circuits accumulate evidence towards a decision. Fitting models of decision-making theory to neural activity helps answer this question, but current approaches limit…

We introduce estimatable variation neural networks (EVNNs), a class of neural networks that allow a computationally cheap estimate on the $BV$ norm motivated by the space $BMV$ of functions with bounded M-variation. We prove a universal…

We develop a first-principles formalism to compute the distortion to the relic neutrino density field caused by the peculiar motions of large-scale structures. This distortion slows halos down due to dynamical friction, causes a local…

Loss functions play a key role in training superior deep neural networks. In convolutional neural networks (CNNs), the popular cross entropy loss together with softmax does not explicitly guarantee minimization of intra-class variance or…

Nearly all practical neural models for classification are trained using cross-entropy loss. Yet this ubiquitous choice is supported by little theoretical or empirical evidence. Recent work (Hui & Belkin, 2020) suggests that training using…

We propose a general framework for neural network compression that is motivated by the Minimum Description Length (MDL) principle. For that we first derive an expression for the entropy of a neural network, which measures its complexity…

We introduce a differentiable, end-to-end trainable framework for solving pixel-level grouping problems such as instance segmentation consisting of two novel components. First, we regress pixels into a hyper-spherical embedding space so…

We apply two variations of the principle of Minimum Cross Entropy (the Kullback information measure) to fit parameterized probability density models to observed data densities. For an array beamforming problem with P incident narrowband…

We investigate the explanability properties of the recently proposed linear-min-max neural networks. At initialization, they can be interpreted as k-medoids with the infinity norm as a distance. Then, they are trained using subgradient…

Identifying the status of individual network units is critical for understanding the mechanism of convolutional neural networks (CNNs). However, it is still challenging to reliably give a general indication of unit status, especially for…

Cross-entropy loss is a common choice when it comes to multiclass classification tasks and language modeling in particular. Minimizing this loss results in language models of very good quality. We show that it is possible to fine-tune these…

Bayesian neural networks (BNNs) provide a formalism to quantify and calibrate uncertainty in deep learning. Current inference approaches for BNNs often resort to few-sample estimation for scalability, which can harm predictive performance,…