Related papers: Upper Bound on Normalized Maximum Likelihood Codes…
Clustering is a widely used technique with a long and rich history in a variety of areas. However, most existing algorithms do not scale well to large datasets, or are missing theoretical guarantees of convergence. This paper introduces a…
Why do larger language models generalize better? To investigate this question, we develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime, as described by the Chinchilla…
In this work, we study the generalization capability of algorithms from an information-theoretic perspective. It has been shown that the expected generalization error of an algorithm is bounded from above by a function of the relative…
We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with $M \geq 3$ components. Our first main result shows that the population likelihood function has bad local maxima even…
Generalized linear mixed models (GLMM) are used for inference and prediction in a wide range of different applications providing a powerful scientific tool. An increasing number of sources of data are becoming available, introducing a…
A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for…
New upper and lower bounds for the error probability over an erasure channel are provided, making use of Wei's generalized weights, hierarchy and spectra. In many situations the upper and lower bounds coincide and this allows improvement of…
We consider linear rank-metric codes in $\mathbb F_{q^m}^n$. We show that the properties of being MRD (maximum rank distance) and non-Gabidulin are generic over the algebraic closure of the underlying field, which implies that over a large…
We study asymptotic lower and upper bounds for the sizes of constant dimension codes with respect to the subspace or injection distance, which is used in random linear network coding. In this context we review known upper bounds and show…
We consider maximum likelihood estimation of finite mixture of uniform distributions. We prove that maximum likelihood estimator is strongly consistent, if the scale parameters of the component uniform distributions are restricted from…
We derive a linear programming bound on the maximum cardinality of error-correcting codes in the sum-rank metric. Based on computational experiments on relatively small instances, we observe that the obtained bounds outperform all…
We study the estimation error of constrained M-estimators, and derive explicit upper bounds on the expected estimation error determined by the Gaussian width of the constraint set. Both of the cases where the true parameter is on the…
An information-theoretic upper bound on the generalization error of supervised learning algorithms is derived. The bound is constructed in terms of the mutual information between each individual training sample and the output of the…
We consider the problem of universal decoding for arbitrary unknown channels in the random coding regime. For a given random coding distribution and a given class of metric decoders, we propose a generic universal decoder whose average…
This paper investigates and bounds the expected solution quality of combinatorial optimization problems when feasible solutions are chosen at random. Loose general bounds are discovered, as well as families of combinatorial optimization…
Generalized Linear Model (or GLM) extends the ordinary linear regression by linking the mean of the response variable to covariates through appropriate link functions. GLM is widely used in the analysis of datasets arising from diverse…
Over the last decade, a wave of research has characterized the exact asymptotic risk of many high-dimensional models in the proportional regime. Two foundational results have driven this progress: Gaussian universality, which shows that the…
The generalised linear model (GLM) is a very important tool for analysing real data in biology, sociology, agriculture, engineering and many other application domain where the relationship between the response and explanatory variables may…
This paper presents a method to calculate the exact average block error probability of some random code ensembles under maximum-likelihood decoding. The proposed method is applicable to various channels and ensembles. The focus is on both…
Score based learning (SBL) is a promising approach for learning Bayesian networks in the discrete domain. However, when employing SBL in the continuous domain, one is either forced to move the problem to the discrete domain or use metrics…