Related papers: Manifold Trajectories in Next-Token Prediction: Fr…

Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We found MTP's exact…

We present a latent variable model for classification that provides a novel probabilistic interpretation of neural network softmax classifiers. We derive a variational objective to train the model, analogous to the evidence lower bound…

Neural machine translation models usually adopt the teacher forcing strategy for training which requires the predicted sequence matches ground truth word by word and forces the probability of each prediction to approach a 0-1 distribution.…

Among random sampling methods, Markov Chain Monte Carlo algorithms are foremost. Using a combination of analytical and numerical approaches, we study their convergence properties towards the steady state, within a random walk Metropolis…

We investigate how next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text. Our analysis, based on a tractable mathematical model and controlled synthetic data, reveals that NTP…

We consider ordinary differential equations on the unit simplex of $\RR^n$ that naturally occur in population games, models of learning and self reinforced random processes. Generalizing and relying on an idea introduced in \cite{DF11}, we…

This paper formalizes a widely used dynamical class--replicator-mutator dynamics and Price-style selection-and-transmission--and makes explicit the modeling choices (scale, atomic unit, interaction topology, transmission kernel) that…

Feature tracking is a common task in visualization applications, where methods based on topological data analysis (TDA) have successfully been applied in the past for feature definition as well as tracking. In this work, we focus on…

Hamiltonian Monte Carlo (HMC) is a powerful Markov chain Monte Carlo (MCMC) method for performing approximate inference in complex probabilistic models of continuous variables. In common with many MCMC methods, however, the standard HMC…

Markov Chain Monte Carlo (MCMC) algorithms are essential tools in computational statistics for sampling from unnormalised probability distributions, but can be fragile when targeting high-dimensional, multimodal, or complex target…

We consider the topology of simplicial complexes with vertices the points of a random point process and faces determined by distance relationships between the vertices. In particular, we study the Betti numbers of these complexes as the…

Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with…

The smoothing distribution is the conditional distribution of the diffusion process in the space of trajectories given noisy observations made continuously in time. It is generally difficult to sample from this distribution. We use the…

Predicting the exact many-body quantum dynamics of polarons in materials with strong carrier-phonon interactions presents a fundamental challenge, often necessitating one to adopt approximations that sacrifice the ability to predict the…

Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the latent representation space under-constrained, allowing…

If the matter produced in ultrarelativistic heavy-ion collisions reaches thermal equilibrium, its subsequent evolution follows the laws of ideal fluid dynamics. We show that general predictions can be made on this basis alone, irrespective…

We propose a differentiable successive halving method of relaxing the top-k operator, rendering gradient-based optimization possible. The need to perform softmax iteratively on the entire vector of scores is avoided by using a…

Due to the fundamental connection between next-symbol prediction and compression, modern predictive models, such as large language models (LLMs), can be combined with entropy coding to achieve compression rates that surpass those of…

Large language models (LLMs) trained on next-token prediction (NTP) paradigm have demonstrated powerful capabilities. However, the existing NTP paradigm contains several limitations, particularly related to planned task complications and…

Bound-bound transitions can occur when localized atomic orbitals are thermally depleted, allowing excitations that would otherwise be forbidden at zero temperature. We predict signatures of bound-bound transitions in x-ray Thomson…