Related papers: Asymptotic Log-loss of Prequential Maximum Likelih…
When in a full exponential family the maximum likelihood estimate (MLE) does not exist, the MLE may exist in the Barndorff-Nielsen completion of the family. We propose a practical algorithm for finding the MLE in the completion based on…
In this paper we provide a method to obtain tight lower bounds on the minimum redundancy achievable by a Huffman code when the probability distribution underlying an alphabet is only partially known. In particular, we address the case where…
Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that…
We establish an equivalence between two important random ensembles of linear codes: random linear codes (RLCs) and random Reed-Solomon (RS) codes. Specifically, we show that these models exhibit identical behavior with respect to key…
In this work, we address the question of the largest rate of linear subcodes of Reed-Muller (RM) codes, all of whose codewords respect a runlength-limited (RLL) constraint. Our interest is in the $(d,\infty)$-RLL constraint, which mandates…
Let $S_n$ be the sum of independent random variables with distribution $F$. Under the assumption that $-\log(1-F(x))$ is slowly varying, conditions for $$ \lim_{n\to\infty}\sup_{s\ge t_n}\left|{P[S_n>s]\over n(1-F(s))}-1\right| =0 $$ are…
We first give a construction of binary $t_1$-deletion-$t_2$-insertion-burst correcting codes with redundancy at most $\log(n)+(t_1-t_2-1)\log\log(n)+O(1)$, where $t_1\ge 2t_2$. Then we give an improved construction of binary codes capable…
Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that its data preferences dynamically change, indicating the need…
Recent multimodal large language models (MLLMs) increasingly integrate multiple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. However, we show…
In this work, we investigate the problem of constructing codes capable of correcting two deletions. In particular, we construct a code that requires redundancy approximately 8 log n + O(log log n) bits of redundancy, where n is the length…
A systematic, comparative investigation into the effects of low-quality data reveals a stark spectrum of robustness across modern probabilistic models. We find that autoregressive language models, from token prediction to…
Parameters of LDPC codes, such as minimum distance, stopping distance, stopping redundancy, girth of the Tanner graph, and their influence on the frame error rate performance of the BP, ML and near-ML decoding over a BEC and an AWGN channel…
Consider a set of categorical variables $\mathcal{P}$ where at least one, denoted by $Y$, is binary. The log-linear model that describes the counts in the resulting contingency table implies a specific logistic regression model, with the…
This paper considers the maximum likelihood estimation of factor models of high dimension, where the number of variables (N) is comparable with or even greater than the number of observations (T). An inferential theory is developed. We…
We consider the lossless compression bound of any individual data sequence. If we fit the data by a parametric model, the entropy quantity $nH({\hat \theta}_n)$ obtained by plugging in the maximum likelihood estimate is an underestimate of…
We prove the existence of codebooks for d-semifaithful lossy compression that are simultaneously universal with respect to both the class of finite-alphabet memoryless sources and the class of all bounded additive distortion measures. By…
In this paper, we study the risk bounds for samples independently drawn from an infinitely divisible (ID) distribution. In particular, based on a martingale method, we develop two deviation inequalities for a sequence of random variables of…
A quantitative theory on the construction and the evolution of the genetic code is proposed. Through introducing the concept of mutational deterioration (MD) and developing a theoretical formalism on MD minimization we have proved: 1, the…
Let $(x_n)_{n=1}^\infty$ be a sequence of integers. We study the number variance of dilations $(\alpha x_n)_{n=1}^\infty$ modulo 1 in intervals of length $S$, and establish pseudorandom (Poissonian) behavior for Lebesgue-almost all $\alpha$…
This article precisely defines huge proofs within the system of Natural Deduction for the Minimal implicational propositional logic \mil. This is what we call an unlimited family of super-polynomial proofs. We consider huge families of…