Related papers: On Triangular Inequality of the Discounted Least I…
We propose an entropy-based information measure, namely the Discounted Least Information Theory of Entropy (DLITE), which not only exhibits important characteristics expected as an information measure but also satisfies conditions of a…
We introduce a new information-theoretic formulation of quantum measurement uncertainty relations, based on the notion of relative entropy between measurement probabilities. In the case of a finite-dimensional system and for any approximate…
We present an information-theoretic framework for discrete diffusion models that yields principled estimators of log-likelihood using score-matching losses. Inspired by the I-MMSE identity for the Gaussian setup, we derive analogous results…
Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise or tripletwise similarity constraints while others take advantage of structured similarity…
Formalising the confrontation of opinions (models) to observations (data) is the task of Inferential Statistics. Information Theory provides us with a basic functional, the relative entropy (or Kullback-Leibler divergence), an asymmetrical…
Measures of dependence among variables, and measures of information content and shared information have become valuable tools of multi-variable data analysis. Information measures, like marginal entropies, mutual and multi-information, have…
Recently, substantial research efforts in Deep Metric Learning (DML) focused on designing complex pairwise-distance losses, which require convoluted schemes to ease optimization, such as sample mining or pair weighting. The standard…
We present numerical results for various information theoretic properties of the square lattice Ising model. First, using a bond propagation algorithm, we find the difference $2H_L(w) - H_{2L}(w)$ between entropies on cylinders of finite…
In many problems in data mining and machine learning, data items that need to be clustered or classified are not points in a high-dimensional space, but are distributions (points on a high dimensional simplex). For distributions, natural…
How much one has learned from an experiment is quantifiable by the information gain, also known as the Kullback-Leibler divergence. The narrowing of the posterior parameter distribution $P(\theta|D)$ compared with the prior parameter…
We proposed a Least Information theory (LIT) to quantify meaning of information in probability distribution changes, from which a new information retrieval model was developed. We observed several important characteristics of the proposed…
We consider an extension of $\epsilon$-entropy to a KL-divergence based complexity measure for randomized density estimation methods. Based on this extension, we develop a general information-theoretical inequality that measures the…
Minimum divergence estimators provide a natural choice of estimators in a statistical inference problem. Different properties of various families of these divergence measures such as Hellinger distance, power divergence, density power…
Kullback-Leibler relative-entropy has unique properties in cases involving distributions resulting from relative-entropy minimization. Tsallis relative-entropy is a one parameter generalization of Kullback-Leibler relative-entropy in the…
In this paper we introduce a nonextensive quantum information theoretic measure which may be defined between any arbitrary number of density matrices, and we analyze its fundamental properties in the spectral graph-theoretic framework.…
The concept of distinguishability lies at the heart of quantum information theory. We introduce \textit{left-right relative entropy} as a quantitative measure of distinguishability within the space of boundary states in two-dimensional…
The distance between the true and numerical solutions in some metric is considered as the discretization error magnitude. If error magnitude ranging is known, the triangle inequality enables the estimation of the vicinity of the approximate…
Kullback-Leibler relative-entropy, in cases involving distributions resulting from relative-entropy minimization, has a celebrated property reminiscent of squared Euclidean distance: it satisfies an analogue of the Pythagoras' theorem. And…
Rooted trees with probabilities are used to analyze properties of a variable length code. A bound is derived on the difference between the entropy rates of the code and a memoryless source. The bound is in terms of normalized informational…
Our capacity to process information depends on the computational power at our disposal. Information theory captures our ability to distinguish states or communicate messages when it is unconstrained with unrivaled beauty and elegance. For…