Related papers: Asymptotic Log-loss of Prequential Maximum Likelih…
We revisit the likelihood ratio between a pretrained large language model (LLM) and its finetuned variant as a criterion for out-of-distribution (OOD) detection. The intuition behind such a criterion is that, the pretrained LLM has the…
For a sample of Exponentially distributed durations we aim at point estimation and a confidence interval for its parameter. A duration is only observed if it has ended within a certain time interval, determined by a Uniform distribution.…
We consider three matrix models of order 2 with one random entry $\epsilon$ and the other three entries being deterministic. In the first model, we let $\epsilon\sim\textrm{Bernoulli}\left(\frac{1}{2}\right)$. For this model we develop a…
We study the performance of machine learning binary classification techniques in terms of error probabilities. The statistical test is based on the Data-Driven Decision Function (D3F), learned in the training phase, i.e., what is…
Determining the largest size, or equivalently finding the lowest redundancy, of q-ary codes for given length and minimum distance is one of the central and fundamental problems in coding theory. Inspired by the construction of…
A maximum likelihood based model selection of discrete Bayesian networks is considered. The model selection is performed through scoring function $S$, which, for a given network $G$ and $n$-sample $D_n$, is defined to be the maximum…
A famous result by Hammersley and Versik-Kerov states that the length $L_n$ of the longest increasing subsequence among $n$ iid continuous random variables grows like $2\sqrt{n}$. We investigate here the asymptotic behavior of $L_n$ for…
We use a simple construction called `recursive subproducts' (that is known to yield good codes of lengths $n^m$, $n \geq 3$) to identify a family of codes sandwiched between first-order and second-order Reed-Muller (RM) codes. These codes…
Correcting insertions/deletions as well as substitution errors simultaneously plays an important role in DNA-based storage systems as well as in classical communications. This paper deals with the fundamental task of constructing codes that…
Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters. We study how much of this decoder capacity is actually needed for speech tasks. Across two LLM…
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One…
The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual…
The Minimum Description Length principle for online sequence estimation/prediction in a proper learning setup is studied. If the underlying model class is discrete, then the total expected square loss is a particularly interesting…
A Lagrange-dual (Gallager-style) lower bound is derived for the error exponent function of the typical random code (TRC) pertaining to the i.i.d. random coding ensemble and mismatched stochastic likelihood decoding. While the original…
Motivated by applications in in-vivo DNA storage, we study codes for correcting duplications. A reverse-complement duplication of length $k$ is the insertion of the reversed and complemented copy of a substring of length $k$ adjacent to its…
Real-world networks are rarely static. Recently, there has been increasing interest in both network growth and network densification, in which the number of edges scales superlinearly with the number of nodes. Less studied but equally…
This study deals with the problem of outliers in ordinal response model, which is a regression on ordered categorical data as the response variable. ``Outlier" means that the combination of ordered categorical data and its covariates is…
In this paper we consider the class of anti-uniform Huffman codes and derive tight lower and upper bounds on the average length, entropy, and redundancy of such codes in terms of the alphabet size of the source. The Fibonacci distributions…
We study two types of probability measures on the set of integer partitions of $n$ with at most $m$ parts. The first one chooses the random partition with a chance related to its largest part only. We then obtain the limiting distributions…
By a classical result of Weyl, for any increasing sequence $(n_k)_{k \geq 1}$ of integers the sequence of fractional parts $(\{n_k x\})_{k \geq 1}$ is uniformly distributed modulo 1 for almost all $x \in [0,1]$. Except for a few special…