Related papers: Deformed Statistics Formulation of the Information…
Mutual Information (MI) is a crucial measure for capturing dependencies between variables, but exact computation is challenging in high dimensions with intractable likelihoods, impacting accuracy and robustness. One idea is to use an…
Different quantities that go by the name of entropy are used in variational principles to infer probability distributions from limited data. Shore and Johnson showed that maximizing the Boltzmann- Gibbs form of the entropy ensures that…
This paper proposes an information-based inference method for partially identified parameters in incomplete models that is valid both when the model is correctly specified and when it is misspecified. Key features of the method are: (i) it…
The information bottleneck (IB) method aims to find compressed representations of a variable $X$ that retain the most relevant information about a target variable $Y$. We show that for a wide family of distributions -- namely, when $Y$ is…
In this paper, we study a remote source coding scenario in which binary phase shift keying (BPSK) modulation sources are corrupted by additive white Gaussian noise (AWGN). An intermediate node, such as a relay, receives these observations…
Rate-distortion theory provides bounds for compressing data produced by an information source to a specified encoding rate that is strictly less than the source's entropy. This necessarily entails some loss, or distortion, between the…
Encoding only the task-related information from the raw data, \ie, disentangled representation learning, can greatly contribute to the robustness and generalizability of models. Although significant advances have been made by regularizing…
We propose a new GAN-based unsupervised model for disentangled representation learning. The new model is discovered in an attempt to utilize the Information Bottleneck (IB) framework to the optimization of GAN, thereby named IB-GAN. The…
The generalized Kullback-Leibler divergence (K-Ld) in Tsallis statistics [constrained by the additive duality of generalized statistics (dual generalized K-Ld)] is here reconciled with the theory of Bregman divergences for expectations…
Time Series Imputation (TSI), which aims to recover missing values in temporal data, remains a fundamental challenge due to the complex and often high-rate missingness in real-world scenarios. Existing models typically optimize the…
Generalizing the group structure of the Euclidean space, we construct a Riemannian metric on the deformed set \ $\mathbb{R}^n_q$ \ induced by the Tsallis entropy composition property. We show that the Tsallis entropy is a "hyperbolic…
The Gibbs paradox is a conventional paradox in classical statistical mechanics, typically resolved by invoking quantum indistinguishability through the 1/N! correction. In this letter, we present a resolution within classical ensemble…
In the world of generalized entropies---which, for example, play a role in physical systems with sub- and super-exponential phasespace growth per degree of freedom---there are two ways for implementing constraints in the maximum entropy…
The framework of non-extensive statistical mechanics, proposed by Tsallis, has been used to describe a variety of systems. The non-extensive statistical mechanics is usually introduced in a formal way, thus simple models exhibiting some…
The aim of the present paper is to present a careful and accessible discussion of the formal aspects of Boltzmann-Gibbs and Tsallis entropies. We begin with a brief overview of Boltzmann-Gibbs entropy, highlighting its main properties and…
Bidirectional language models have better context understanding and perform better than unidirectional models on natural language understanding tasks, yet the theoretical reasons behind this advantage remain unclear. In this work, we…
The Tsallis entropy and Fisher information entropy (matrix) are very important quantities expressing information measures in nonextensive systems. Stationary and dynamical properties of the information entropies have been investigated in…
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This…
We show how Fisher's information already known particular character as the fundamental information geometric object which plays the role of a metric tensor for a statistical differential manifold, can be derived in a relatively easy manner…
We provide numerical indications of the $q$-generalised central limit theorem that has been conjectured (Tsallis 2004) in nonextensive statistical mechanics. We focus on $N$ binary random variables correlated in a {\it scale-invariant} way.…