Related papers: Deformed Statistics Formulation of the Information…
In density estimation task, maximum entropy model (Maxent) can effectively use reliable prior information via certain constraints, i.e., linear constraints without empirical parameters. However, reliable prior information is often…
Nonadditive Tsallis $q$-statistics has successfully been applied for a plethora of systems in natural sciences and other branches of knowledge. Nevertheless, its foundations have been severely criticised by some authors based on the…
Few-shot learning (FSL), which aims to classify unseen classes with few samples, is challenging due to data scarcity. Although various generative methods have been explored for FSL, the entangled generation process of these methods…
Tsallis has proposed a generalization of Boltzmann-Gibbs thermostatistics by introducing a family of generalized nonextensive entropy functionals with a single parameter $q$. These reduce to the extensive Boltzmann-Gibbs form for $q=1$, but…
We tackle a common domain adaptation setting in causal systems. In this setting, the target variable is observed in the source domain but is entirely missing in the target domain. We aim to impute the target variable in the target domain…
Whereas Shannon entropy is related to the growth rate of multinomial coefficients, we show that the quadratic entropy (Tsallis 2-entropy) is connected to their $q$-deformation; when $q$ is a prime power, these $q$-multinomial coefficients…
Information Bottleneck (IB) is a widely used framework that enables the extraction of information related to a target random variable from a source random variable. In the objective function, IB controls the trade-off between data…
The Lagrangian technique of Niven (2004, Physica A, 334(3-4): 444) is used to determine the constrained forms of the Tsallis entropy function - i.e. Lagrangian functions in which the probabilities of each state are independent - for each…
We have discussed dynamical properties of the Tsallis entropy and the generalized Fisher information in nonextensive systems described by the Langevin model subjected to additive and multiplicative noise. Analytical expressions for the…
Integration of data from multiple omics techniques is becoming increasingly important in biomedical research. Due to non-uniformity and technical limitations in omics platforms, such integrative analyses on multiple omics, which we refer to…
E.T. Jaynes, originator of the maximum entropy interpretation of statistical mechanics, emphasized that there is an inevitable trade-off between the conflicting requirements of robustness and accuracy for any inferencing algorithm. This is…
Deep neural networks (DNNs) have achieved significant success in various applications with large-scale and balanced data. However, data in real-world visual recognition are usually long-tailed, bringing challenges to efficient training and…
We study the problem of distributed information bottleneck, in which multiple encoders separately compress their observations in a manner such that, collectively, the compressed signals preserve as much information as possible about another…
We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or…
Whenever communication takes place to fulfil a goal, an effective way to encode the source data to be transmitted is to use an encoding rule that allows the receiver to meet the requirements of the goal. A formal way to identify the…
A principled framework to generalize variational perturbation approximations (VPA's) formulated within the ambit of the nonadditive statistics of Tsallis statistics, is introduced. This is accomplished by operating on the terms constituting…
The non-extensive canonical ensemble theory is reconsidered with the method of Lagrange multipliers by maximizing Tsallis entropy, with the constraint that the normalized term of Tsallis' $q-$average of physical quantities, the sum $\sum…
We introduce a bottleneck method for learning data representations based on information deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method efficiently. The…
The nonextensive statistical ensembles are revisited for the complex systems with long-range interactions and long-range correlations. An approximation, the value of nonextensive parameter (1-q) is assumed to be very tiny, is adopted for…
We propose a unified information-geometric framework that formalizes understanding in learning as a trade-off between informativeness and geometric simplicity. An encoder phi is evaluated by U(phi) = I(phi(X); Y) - beta * C(phi), where…