Related papers: Simple and Sharp Generalization Bounds via Lifting
Compute-efficient training of language models has become an important issue. We consider data pruning for data-efficient training of LLMs. In this work, we consider a data pruning method based on information entropy. We propose that the…
Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these…
We investigate query-to-communication lifting theorems for models related to the quantum adversary bounds. Our results are as follows: 1. We show that the classical adversary bound lifts to a lower bound on randomized communication…
Often in the analysis of first-order methods, assuming the existence of a quadratic growth bound (a generalization of strong convexity) facilitates much stronger convergence analysis. Hence the analysis is done twice, once for the general…
We study an entropy measure for quantum systems that generalizes the von Neumann entropy as well as its classical counterpart, the Gibbs or Shannon entropy. The entropy measure is based on hypothesis testing and has an elegant formulation…
Simple Exponential Smoothing is a classical technique used for smoothing time series data by assigning exponentially decreasing weights to past observations through a recursive equation; it is sometimes presented as a rule of thumb…
We present a simple and clear foundation for finite inference that unites and significantly extends the approaches of Kolmogorov and Cox. Our approach is based on quantifying lattices of logical statements in a way that satisfies general…
One of the big challenges in the development of probabilistic relational (or probabilistic logical) modeling and learning frameworks is the design of inference techniques that operate on the level of the abstract model representation…
The paper addresses the problem of sampling discretization of integral norms of elements of finite-dimensional subspaces satisfying some conditions. We prove sampling discretization results under two standard kinds of assumptions --…
We introduce an information-theoretic framework for smooth structures on topological manifolds, replacing coordinate charts with small-scale entropy data of local probability probes. A concise set of axioms identifies admissible coordinate…
We build a solvability theory of elliptic boundary-value problems in normed Sobolev spaces of generalized smoothness for any integrability exponent $p>1$. The smoothness is given by a number parameter and a supplementary function parameter…
This paper is concerned with the study of constrained statistical learning problems, the unconstrained version of which are at the core of virtually all of modern information processing. Accounting for constraints, however, is paramount to…
This paper reveals a conceptually new connection from information theory to approximation theory via quantum algorithms for entropy estimation. Specifically, we provide an information-theoretic lower bound $\Omega(\sqrt{n})$ on the…
We introduce a generic framework for solving linear programs (LPs) with many constraints $(n \gg d)$ via adaptive sparsification. Our approach provides a principled generalization of the techniques of [Assadi '23] from matching problems to…
The basic disentanglement theorem established by the present authors states that estimates on a weighted geometric mean over (convex) families of functions can be disentangled into quantitatively linked estimates on each family separately.…
This study critically analyses the information-theoretic, axiomatic and combinatorial philosophical bases of the entropy and cross-entropy concepts. The combinatorial basis is shown to be the most fundamental (most primitive) of these three…
Cut-elimination is the bedrock of proof theory with a multitude of applications from computational interpretations to proof analysis. It is also the starting point for important meta-theoretical investigations including decidability,…
In many astronomical problems one often needs to determine the upper and/or lower boundary of a given data set. An automatic and objective approach consists in fitting the data using a generalised least-squares method, where the function to…
This paper provides an elementary, self-contained analysis of diffusion-based sampling methods for generative modeling. In contrast to existing approaches that rely on continuous-time processes and then discretize, our treatment works…
Many economic theory models incorporate finiteness assumptions that, while introduced for simplicity, play a real role in the analysis. We provide a principled framework for scaling results from such models by removing these finiteness…