Related papers: Three tutorial lectures on entropy and counting
We construct the generalized entropy optimized by a given arbitrary statistical distribution with a finite linear expectation value of a random quantity of interest. This offers, via the maximum entropy principle, a unified basis for a…
The average uncertainty associated with words is an information-theoretic concept at the heart of quantitative and computational linguistics. The entropy has been established as a measure of this average uncertainty - also called average…
In this tutorial we review the essential arguments behing entropic inference. We focus on the epistemological notion of information and its relation to the Bayesian beliefs of rational agents. The problem of updating from a prior to a…
Informational entropy is often identified as physical entropy. This is surprising because the two quantities are differently defined and furthermore the former is a subjective quantity while the latter is an objective one. We describe the…
The ideas of the constraint-based statistics are introduced. The axiom and six theorems of the constraint-based statistics are established.
In order to undestand how the complex concept of entropy emerged,we propose a trip towards the past reviewing the works of Clausius, Boltzmann, Gibbs and Planck. In particular, since the Gibbs's work is not very well known, we present a…
Notions of topological free entropy and of free capacity are introduced in the $C^*$-algebra context. Basic properties, basic problems and connections to potential theory and random matrix theory are discussed.
We present two new estimators for estimating the entropy of absolutely continuous random variables. Some properties of them are considered, specifically consistency of the first is proved. The introduced estimators are compared with the…
This work considers the notion of random tensors and reviews some fundamental concepts in statistics when applied to a tensor based data or signal. In several engineering fields such as Communications, Signal Processing, Machine learning,…
In this paper we introduce several natural definitions of asymptotic independence of two sequences of random elements. We discuss their basic properties, some simple connections between them and connections with properties of weak…
Mixture models have been around for over 150 years, as an intuitively simple and practical tool for enriching the collection of probability distributions available for modelling data. In this chapter we describe the basic ideas of the…
This tutorial investigates the convergence of statistical mechanics and learning theory, elucidating the potential enhancements in machine learning methodologies through the integration of foundational principles from physics. The tutorial…
Information theory, though originally developed for communications engineering, provides mathematical tools with broad applications across science. These tools characterize the fundamental limits of data compression and transmission in the…
In this paper, we investigate and compare two well-developed definitions of entropy relevant for describing the dynamics of isolated quantum systems: bipartite entanglement entropy and observational entropy. In a model system of interacting…
Transfer entropy is a widely used measure for quantifying directed information flows in complex systems. While the challenges of estimating transfer entropy for continuous data are well known, it has two major shortcomings for data of…
We consider in this paper the problem of computing the entropy of a braid. We recall its definition and construct, for each braid, a sequence of real numbers, whose limit is its entropy. We state one conjecture about the convergence speed,…
This essay advocates the view that any problem that has a meaningful empirical content, can be formulated in constructive, more definitely, finite terms. We consider combinatorial models of dynamical systems and approaches to statistical…
Random tilings are interesting as idealizations of atomistic models of quasicrystals and for their connection to problems in combinatorics and algorithms. Of particular interest is the tiling entropy density, which measures the relation of…
We define a general notion of entropy in elementary, algebraic terms. Based on that, weak forms of a scalar product and a distance measure are derived. We give basic properties of these quantities, generalize the Cauchy-Schwarz inequality,…
Entropy is a fundamental property of a repertoire. Here, we present an efficient algorithm to estimate the entropy of types with the help of Zhang's estimator. The algorithm takes advantage of the fact that the number of different…