Related papers: On the Kolmogorov Complexity of Binary Classifiers
This paper addresses the challenge of identifying a minimal subset of discrete, independent variables that best predicts a binary class. We propose an efficient iterative method that sequentially selects variables based on which one…
The information in an individual finite object (like a binary string) is commonly measured by its Kolmogorov complexity. One can divide that information into two parts: the information accounting for the useful regularity present in the…
In this paper order estimates for the Kolmogorov widths of weighted Sobolev classes on a multi-dimensional domain are obtained. The classes are defined by conditions on the highest-order derivatives and the derivative of order zero.
The main result is that: function descriptions are not made equal, and they can be categorised in at least two categories using various computational methods for function evaluation. The result affects Kolmogorov complexity and Random…
Li, Chen, Li, Ma, and Vit\'anyi (2004) introduced a similarity metric based on Kolmogorov complexity. It followed work by Shannon in the 1950s on a metric based on entropy. We define two computable similarity metrics, analogous to the…
Mutual information I in infinite sequences (and in their finite prefixes) is essential in theoretical analysis of many situations. Yet its right definition has been elusive for a long time. I address it by generalizing Kolmogorov Complexity…
In this paper we obtain asymptotic estimates of Kolmogorov and linear widths of the weighted Besov classes with singularity at the origin. In addition, estimates of Kolmogorov and linear widths of finite-dimensional balls in a mixed norm…
While the optimal sample complexity of binary classification in terms of the VC dimension is well-established, determining the optimal sample complexity of multiclass classification has remained open. The appropriate complexity parameter…
Kolmogorov (1965) defined the complexity of a string $x$ as the minimal length of a program generating $x$. Obviously this definition depends on the choice of the programming language. Kolmogorov noted that there exist \emph{optimal}…
For a measurable map $T$ and a sequence of $T$-invariant probability measures $\mu_n$ that converges in some sense to a $T$-invariant probability measure $\mu$, an estimate from below for the Kolmogorov--Sinai entropy of $T$ with respect to…
Inspired by logistic regression, we introduce a regression model for data tuples consisting of a binary response and a set of covariates residing in a metric space without vector structures. Based on the proposed model we also develop a…
We study the minimax sample complexity of multicalibration in the batch setting. A learner observes $n$ i.i.d. samples from an unknown distribution and must output a (possibly randomized) predictor whose population multicalibration error,…
We bound the future loss when predicting any (computably) stochastic sequence online. Solomonoff finitely bounded the total deviation of his universal predictor M from the true distribution m by the algorithmic complexity of m. Here we…
In classical statistical learning theory, one of the most well studied problems is that of binary classification. The information-theoretic sample complexity of this task is tightly characterized by the Vapnik-Chervonenkis (VC) dimension. A…
Many binary classification problems minimize misclassification above (or below) a threshold. We show that instances of ranking problems, accuracy at the top or hypothesis testing may be written in this form. We propose a general framework…
Consider a multi-class labelling problem, where the labels can take values in $[k]$, and a predictor predicts a distribution over the labels. In this work, we study the following foundational question: Are there notions of multi-class…
The even online Kolmogorov complexity of a string $x = x_1 x_2 \cdots x_{n}$ is the minimal length of a program that for all $i\le n/2$, on input $x_1x_3 \cdots x_{2i-1}$ outputs $x_{2i}$. The odd complexity is defined similarly. The sum of…
Kolmogorov argued that the concept of information exists also in problems with no underlying stochastic model (as Shannon's information representation) for instance, the information contained in an algorithm or in the genome. He introduced…
The Kolmogorov entropy allows to split the dynamical systems that have equivalent continuous spectrum into non-isomorphic subclasses. In this paper we make an attempt to generalise the concept of entropy that will allow to split the systems…
We show that the mutual information, in the sense of Kolmogorov complexity, of any pair of strings $x$ and $y$ is equal, up to logarithmic precision, to the length of the longest shared secret key that two parties, one having $x$ and the…