Related papers: On Enumerating Distributions for Associated Vector…
In many high-impact applications, it is important to ensure the quality of output of a machine learning algorithm as well as its reliability in comparison with the complexity of the algorithm used. In this paper, we have initiated a…
Maximum entropy models are increasingly being used to describe the collective activity of neural populations with measured mean neural activities and pairwise correlations, but the full space of probability distributions consistent with…
The phenomenon of entropy concentration provides strong support for the maximum entropy method, MaxEnt, for inferring a probability vector from information in the form of constraints. Here we extend this phenomenon, in a discrete setting,…
We consider the problem of learning a target probability distribution over a set of $N$ binary variables from the knowledge of the expectation values (with this target distribution) of $M$ observables, drawn uniformly at random. The space…
We introduce a new measure of interdependence among the components of a random vector along the main diagonal of the vector copula, i.e. along the line $u_{1}=\ldots=u_{J}$, for $\left(u_{1},\ldots,u_{J}\right)\in\left[0,1\right]^{J}$. Our…
We study the problem of estimating a distribution over a finite alphabet from an i.i.d. sample, with accuracy measured in relative entropy (Kullback-Leibler divergence). While optimal bounds on the expected risk are known, high-probability…
We introduce a family of scale-invariant entropy statistics derived from logarithmically aggregated distance distributions of point processes, with prime numbers serving as a motivating example. The construction associates to each finite…
Existing training criteria in automatic speech recognition(ASR) permit the model to freely explore more than one time alignments between the feature and label sequences. In this paper, we use entropy to measure a model's uncertainty, i.e.…
In many problems in data mining and machine learning, data items that need to be clustered or classified are not points in a high-dimensional space, but are distributions (points on a high dimensional simplex). For distributions, natural…
We study the relative entropy between the empirical estimate of a discrete distribution and the true underlying distribution. If the minimum value of the probability mass function exceeds an $\alpha > 0$ (i.e. when the true underlying…
For various purposes and, in particular, in the context of data compression, a graph can be examined at three levels. Its structure can be described as the unlabeled version of the graph; then the labeling of its structure can be added; and…
The goal of this paper is to develop an estimate for the entropy of random long-range correlated symbolic sequences with elements belonging to a finite alphabet. As a plausible model, we use the high-order additive stationary ergodic Markov…
A common statistical situation concerns inferring an unknown distribution Q(x) from a known distribution P(y), where X (dimension n), and Y (dimension m) have a known functional relationship. Most commonly, n<m, and the task is relatively…
Mixture distributions are a workhorse model for multimodal data in information theory, signal processing, and machine learning. Yet even when each component density is simple, the differential entropy of the mixture is notoriously hard to…
Problems of probabilistic inference and decision making under uncertainty commonly involve continuous random variables. Often these are discretized to a few points, to simplify assessments and computations. An alternative approximation is…
In this paper, we investigate the problem of classifying feature vectors with mutually independent but non-identically distributed elements. First, we show the importance of this problem. Next, we propose a classifier and derive an…
One way of getting insight into non-Gaussian measures, posed on infinite dimensional Hilbert spaces, is to first obtain best fit Gaussian approximations, which are more amenable to numerical approximation. These Gaussians can then be used…
We consider the set of finite sequences of length n over a finite or countable alphabet C. We consider the function which associate each given sequence with the size of the maximum overlap with a (shifted) copy of itself. We compute the…
We explore the relation between entanglement entropy of quantum many body systems and the distribution of corresponding, properly selected, observables. Such a relation is necessary to actually measure the entanglement entropy. We show that…
We consider shift spaces in which elements of the alphabet may overlap nontransitively. We define a notion of entropy for such spaces, give several techniques for computing lower bounds for it, and show that it is equal to a limit of…