Related papers: Generative Models with ELBOs Converging to Entropy…

State-of-the-art language generation models can degenerate when applied to open-ended generation problems such as text completion, story generation, or dialog modeling. This degeneration usually shows up in the form of incoherence, lack of…

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an indispensable paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard policy optimization methods, such as Group Relative Policy…

Lower bounds for the R\'enyi entropies of sums of independent random variables taking values in cyclic groups of prime order under permutations are established. The main ingredients of our approach are extended rearrangement inequalities in…

In this paper, we extend the concept of generalized entropy to uniform spaces, allowing computations beyond metrizable settings. We apply this to parabolic dynamics - systems with a unique fixed point uniformly attracting all compact…

There is accumulating evidence in the literature that stability of learning algorithms is a key characteristic that permits a learning algorithm to generalize. Despite various insightful results in this direction, there seems to be an…

Bayesian Neural Networks (BNNs) are trained to optimize an entire distribution over their weights instead of a single set, having significant advantages in terms of, e.g., interpretability, multi-task learning, and calibration. Because of…

Entropy production (EP) is a key quantity in thermodynamics, and yet measuring EP has remained a challenging task. Here we introduce an EP estimator, called multidimensional entropic bound (MEB), utilizing an ensemble of trajectories…

In this paper, we improve the known estimates for the invariance entropy of a nonlinear control system. For sets of complete approximate controllability we derive an upper bound in terms of Lyapunov exponents and for uniformly hyperbolic…

We look into the task of \emph{generalizing} word embeddings: given a set of pre-trained word vectors over a finite vocabulary, the goal is to predict embedding vectors for out-of-vocabulary words, \emph{without} extra contextual…

The aim of this paper is to provide several novel upper bounds on the excess risk with a primal focus on classification problems. We suggest two approaches and the obtained bounds are represented via the distribution dependent local…

Entropy serves as a central observable in equilibrium thermodynamics. However, many biological and ecological systems operate far from thermal equilibrium. Here we show that entropy production can characterize the behavior of such…

We present a general framework for deriving entropy production rates (EPRs) in active matter systems driven by non-Gaussian active fluctuations. Employing the probability-flow equivalence technique, we rigorously obtain an entropy…

Thermodynamic extensivity is commonly introduced as a postulate -- the homogeneity of degree one in thermodynamic potentials. We provide a constructive derivation of this property from microscopic conditions on the pair potential, without…

The large-time asymptotics of the solutions to a class of degenerate parabolic cross-diffusion systems is analyzed. The equations model the interaction of an arbitrary number of population species in a bounded domain with no-flux boundary…

Exposure bias refers to the train-test discrepancy that seemingly arises when an autoregressive generative model uses only ground-truth contexts at training time but generated ones at test time. We separate the contributions of the model…

The Bayesian approach to solving inverse problems relies on the choice of a prior. This critical ingredient allows the formulation of expert knowledge or physical constraints in a probabilistic fashion and plays an important role for the…

Selective inference methods are developed for group lasso estimators for use with a wide class of distributions and loss functions. The method includes the use of exponential family distributions, as well as quasi-likelihood modeling for…

Autoregressive language models have demonstrated a remarkable ability to extract latent structure from text. The embeddings from large language models have been shown to capture aspects of the syntax and semantics of language. But what…

Large Reasoning Models are typically trained via reinforcement learning from verifiable rewards (RLVR). However, existing approaches adopt fixed weights for positive and negative samples, and the conclusions hardly generalize to open-ended…

Consensus-based optimization (CBO) is a powerful and versatile zero-order multi-particle method designed to provably solve high-dimensional global optimization problems, including those that are genuinely nonconvex or nonsmooth. The method…