Related papers: Deformed Statistics Formulation of the Information…
Normalization is fundamental to deep learning, but existing approaches such as BatchNorm, LayerNorm, and RMSNorm are variance-centric by enforcing zero mean and unit variance, stabilizing training without controlling how representations…
The equipartition theorem states that inverse temperature equals the log-derivative of the density of states. This relation can be generalized by introducing a proportionality factor involving an increasing positive function phi(x). It is…
Deep learning has become the most powerful machine learning tool in the last decade. However, how to efficiently train deep neural networks remains to be thoroughly solved. The widely used minibatch stochastic gradient descent (SGD) still…
Tsallis' non-extensive entropy is extended to incorporate the dependence on affinities between the microstates of a system. At the core of our construction of the extended entropy ($\mathcal{H}$) is the concept of the effective number of…
An unified thermodynamical framework based in the use of a generalized Massieu-Planck thermodynamic potential is proposed and a new formulation of Boltzmann-Gibbs Statistical Mechanics is established. Under this philosophy a generalization…
Graph data, essential in fields like knowledge representation and social networks, often involves large networks with many nodes and edges. Transmitting these graphs can be highly inefficient due to their size and redundancy for specific…
Tsallis and R\'{e}nyi entropies, which are monotone transformations of each other, are deformations of the celebrated Shannon entropy. Maximization of these deformed entropies, under suitable constraints, leads to the $q$-exponential family…
We derive a novel version of information-disturbance theorems for mutually unbiased observables. We show that the information gain by Eve inevitably makes the outcomes by Bob in the conjugate basis not only erroneous but random.
Selecting an optimal subset of features or instances under an information theoretic criterion has become an effective preprocessing strategy for reducing data complexity while preserving essential information. This study investigates two…
We study the informational underpinnings of thermodynamics and statistical mechanics, using an abstract framework, general probabilistic theories, capable of describing arbitrary physical theories. This allows one to abstract the…
Statistical inference is considered for variables of interest, called primary variables, when auxiliary variables are observed along with the primary variables. We consider the setting of incomplete data analysis, where some primary…
We review with a tutorial scope the information theory foundations of quantum statistical physics. Only a small proportion of the variables that characterize a system at the microscopic scale can be controlled, for both practical and…
Contrastive learning is effective for aligning paired views or modalities, but alignment beyond two modalities remains non-trivial and comparatively underexplored. Pairwise CLIP-style losses decompose multi-modal alignment into independent…
This study comes as a timely response to mounting criticism of the information bottleneck (IB) theory, injecting fresh perspectives to rectify misconceptions and reaffirm its validity. Firstly, we introduce an auxiliary function to…
We discuss the generalized von Neumann (Tsallis) entropy and the generalized Fisher information (GFI) in nonextensive quantum systems, by using the interpolation approximation (IA) which has been shown to yield good results for the quantal…
In order to improve the teaching of the course of statistical physics in universities, in this article we introduce nonextensive statistics, a new statistical theory about complex systems. We study the two modification coefficients a and b…
Based on the form invariance of the structures given by Khinchin's axiomatic foundations of information theory and the pseudoadditivity of the Tsallis entropy indexed by q, the concept of conditional entropy is generalized to the case of…
Learned representations at the level of characters, sub-words, words and sentences, have each contributed to advances in understanding different NLP tasks and linguistic phenomena. However, learning textual embeddings is costly as they are…
By extracting task-relevant information while maximally compressing the input, the information bottleneck (IB) principle has provided a guideline for learning effective and robust representations of the target inference. However, extending…
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural…