Related papers: Hidden information and regularities of information…
By applying the holographic principle that comes from black hole physics and information entropy (Sient) introduced by Claude Shannon, we derive the mechanism of similar patterns or tendencies appearing in evolution processes of different…
The emergence of collective dynamics in neural networks is a mechanism of the animal and human brain for information processing. In this paper, we develop a computational technique using distributed processing elements in a complex network,…
Imitation is an important learning heuristic in animal and human societies. Previous explorations report that the fate of individuals with cooperative strategies is sensitive to the protocol of imitation, leading to a conundrum about how…
We develop an axiomatic reconstruction of thermodynamics based entirely on two primitive components: a description of what aspects of a system are observed and a reference measure that encodes the underlying descriptive convention. These…
"Standard" information theory says nothing about the semantic content of information. Nevertheless, applications such as evolutionary theory demand consideration of precisely this aspect of information, a need that has motivated a largely…
The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between…
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This…
This paper presents a method to explain how the information of each input variable is gradually discarded during the forward propagation in a deep neural network (DNN), which provides new perspectives to explain DNNs. We define two types of…
We present an information geometric analysis of both entropic speeds and entropy production rates arising from geodesic evolution on manifolds parametrized by pure quantum states. In particular, we employ pure states that emerge as outputs…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
Information dynamics is an emerging description of information processing in complex systems which describes systems in terms of intrinsic computation, identifying computational primitives of information storage and transfer. In this paper…
This study investigates entropy's potential for analyzing scientific research patterns across disciplines. Originating from thermodynamics, entropy now measures uncertainty and diversity in information systems. We examine Shannon Entropy,…
I formulate an entropy-rate maximization problem at the observable level for stochastic processes observed through an information-reducing observation map. For a visible stationary law, the map determines an observational fiber of hidden…
We study the large deviations performance, i.e., the exponential decay rate of the error probability, of distributed detection algorithms over random networks. At each time step $k$ each sensor: 1) averages its decision variable with the…
We propose a novel approach framed in terms of information theory and entropy to tackle the issue of conspiracy theories propagation. We start with the report of an event (such as 9/11 terroristic attack) represented as a series of…
Three steps aid in the analysis of selection. First, describe phenotypes by their component causes. Components include genes, maternal effects, symbionts, and any other predictors of phenotype that are of interest. Second, describe fitness…
We try to establish a unified information theoretic approach to learning and to explore some of its applications. First, we define {\em predictive information} as the mutual information between the past and the future of a time series,…
This paper studies the Shannon regime for the random displacement of stationary point processes. Let each point of some initial stationary point process in $\R^n$ give rise to one daughter point, the location of which is obtained by adding…
Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we…
In this paper, we present a new multi-scale information content calculation method based on Shannon information (and Shannon entropy). The original method described by Claude E. Shannon and based on the logarithm of the probability of…