Related papers: Softmax as a Lagrangian-Legendrian Seam

Large language models (LLMs) are pretrained by minimizing the cross-entropy loss for next-token prediction. In this paper, we study whether this optimization strategy can induce geometric structure in the learned model weights and context…

With modern high-dimensional data, complex statistical models are necessary, requiring computationally feasible inference schemes. We introduce Max-and-Smooth, an approximate Bayesian inference scheme for a flexible class of latent Gaussian…

Using convex integration we give a constructive proof of the well-known fact that every continuous curve in a contact $3$-manifold can be approximated by a Legendrian curve.

Large language models (LLMs) have made transformed changes for human society. One of the key computation in LLMs is the softmax unit. This operation is important in LLMs because it allows the model to generate a distribution over possible…

Particles added to a fluid interface can be used as a surface stabilizer in the food, oil and cosmetic industries. As an alternative to rigid particles, it is promising to consider highly deformable particles that can adapt their…

In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax…

Deep Metric Learning (DML) models rely on strong representations and similarity-based measures with specific loss functions. Proxy-based losses have shown great performance compared to pair-based losses in terms of convergence speed.…

We provide two characterizations, one axiomatic and the other neuro-computational, of the dependence of choice probabilities on deadlines, within the widely used softmax representation \[ p_{t}\left( a,A\right) =\dfrac{e^{\frac{u\left(…

Simulating soft matter systems such as the cytoskeleton can enable deep understanding of experimentally observed phenomena. One challenge of modeling such systems is realistic description of the steric repulsion between nearby polymers.…

The Plateau-Rayleigh instability causes the fragmentation of a liquid ligament into smaller droplets. In this study a numerical study of this phenomenon based on a single relaxation time (SRT) pseudo-potential lattice Boltzmann method (LBM)…

Softmax is widely used in neural networks for multiclass classification, gate structure and attention mechanisms. The statistical assumption that the input is normal distributed supports the gradient stability of Softmax. However, when used…

The softmax function is a widely used activation function in the output layers of neural networks, responsible for converting raw scores into class probabilities while introducing essential non-linearity. Implementing Softmax efficiently…

Large language models (LLMs) have brought significant changes to human society. Softmax regression and residual neural networks (ResNet) are two important techniques in deep learning: they not only serve as significant theoretical…

To predict flow behavior of entangled polymer melt, we have developed multiscale simulation composed of Lagrangian fluid particle simulation and coarse-grained polymer dynamics simulation. We have introduced a particle deformation in the…

Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus for this soft-to-hard limit in population squared-loss…

Articles in Marketing and choice literatures have demonstrated the need for incorporating person-level heterogeneity into behavioral models (e.g., logit models for multiple binary outcomes as studied here). However, the logit likelihood…

Multivariate machine learning techniques provide an alternative to the rapidity gap method for event-by-event identification and classification of diffraction in hadron-hadron collisions. Traditionally, such methods assign each event…

A GPU-accelerated version of the lattice Boltzmann method for efficient simulation of soft materials is introduced. Unlike standard approaches, this method reconstructs the distribution functions from available hydrodynamic variables…

Stochastic Gradient Descent (SGD), a widely used optimization algorithm in deep learning, is often limited to converging to local optima due to the non-convex nature of the problem. Leveraging these local optima to improve model performance…

In this work, we investigate the interaction of free and porous media flow by large scale lattice Boltzmann simulations. We study the transport phenomena at the porous interface on multiple scales, i.e., we consider both, computationally…