相关论文: Manifold Trajectories in Next-Token Prediction: Fr…

Using two extremely different models of glass formers in two and three dimensions we demonstrate how to encode the subtle changes in the geometric rearrangement of particles during the scenario of the glass transition. We construct a…

Distribution regression seeks to estimate the conditional distribution of a multivariate response given a continuous covariate. This approach offers a more complete characterization of dependence than traditional regression methods.…

We prove approach to thermal equilibrium for the fully Hamiltonian dynamics of a dynamical Lorentz gas, by which we mean an ensemble of particles moving through a $d$-dimensional array of fixed soft scatterers that each possess an internal…

Diffusion language models are a promising alternative to autoregressive models, yet post-training methods for them largely adapt reward-maximizing objectives. We identify a central failure mode in this setting we call trajectory locking:…

Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were not compute-limited, should we still use decoder-only…

Uncertainty approximation in text classification is an important area with applications in domain adaptation and interpretability. One of the most widely used uncertainty approximation methods is Monte Carlo (MC) Dropout, which is…

We propose a new model for multi-token prediction in transformers, aiming to enhance sampling efficiency without compromising accuracy. Motivated by recent work that predicts the probabilities of subsequent tokens using multiple heads, we…

Text-to-image generation models suffer from alignment problems, where generated images fail to accurately capture the objects and relations in the text prompt. Prior work has focused on improving alignment by refining the diffusion process,…

Evolutionary dynamics in an uncorrelated rugged fitness landscape is studied in the framework of Eigen's molecular quasispecies model. We consider the case of strong selection, which is analogous to the zero temperature limit in the…

A Monte Carlo method for simulating a multi-dimensional diffusion process conditioned on hitting a fixed point at a fixed future time is developed. Proposals for such diffusion bridges are obtained by superimposing an additional guiding…

There are many fields where the transition from diffusive to ballistic motion is important. Here we deal with relaxation processes in nmr in gases. Correlation functions for trajectory variables (position and velocity) valid across this…

Metriplectic dynamical systems consist of a special combination of a Hamiltonian and a (generalized) entropy-gradient flow, such that the Hamiltonian is conserved and entropy is dissipated/produced (depending on a sign convention). It is…

Recent empirical evidence shows that LLM representations encode human-interpretable concepts. Nevertheless, the mechanisms by which these representations emerge remain largely unexplored. To shed further light on this, we introduce a novel…

Mantle convection simulations are an essential tool for understanding how rocky planets evolve. However, the poorly known input parameters to these simulations, the non-linear dependence of transport properties on pressure and temperature,…

We study slow dynamics of particles moving in a matrix of immobile obstacles using molecular dynamics simulations. The glass transition point decreases drastically as the obstacle density increases. At higher obstacle densities, the…

The Gumbel-Max trick is the basis of many relaxed gradient estimators. These estimators are easy to implement and low variance, but the goal of scaling them comprehensively to large combinatorial distributions is still outstanding. Working…

Score-based diffusion models are a highly effective method for generating samples from a distribution of images. We consider scenarios where the training data comes from a noisy version of the target distribution, and present an efficiently…

We present a continuous time generalization of a random walk with complete memory of its history [Phys. Rev. E 70, 045101(R) (2004)] and derive exact expressions for the first four moments of the distribution of displacement when the number…

Langevin diffusion is a powerful method for nonconvex optimization, which enables the escape from local minima by injecting noise into the gradient. In particular, the temperature parameter controlling the noise level gives rise to a…

Recently a new class of Monte Carlo methods, called Time Relaxed Monte Carlo (TRMC), designed for the simulation of the Boltzmann equation close to fluid regimes have been introduced. A generalized Wild sum expansion of the solution is at…