Related papers: Which Similarity-Sensitive Entropy (Sentropy)?
We apply two variations of the principle of Minimum Cross Entropy (the Kullback information measure) to fit parameterized probability density models to observed data densities. For an array beamforming problem with P incident narrowband…
The variance of (relative) surprisal, also known as varentropy, so far mostly plays a role in information theory as quantifying the leading order corrections to asymptotic i.i.d.~limits. Here, we comprehensively study the use of it to…
In a world where more decisions are made using artificial intelligence, it is of utmost importance to ensure these decisions are well-grounded. Neural networks are the modern building blocks for artificial intelligence. Modern neural…
We study the relative R\'enyi entropy (RRE) under local quenches in two-dimensional conformal field theories (CFTs), focusing on rational CFTs (RCFTs) and holographic CFTs. In RCFTs, the RRE evolves as a monotonic function over time,…
High signal to noise ratio (SNR) consistency of model selection criteria in linear regression models has attracted a lot of attention recently. However, most of the existing literature on high SNR consistency deals with model order…
Shannon Entropy has been extensively used for characterizing complexity of time series arising from chaotic dynamical systems and stochastic processes such as Markov chains. However, for short and noisy time series, Shannon entropy performs…
Entropy and relative or cross entropy measures are two very fundamental concepts in information theory and are also widely used for statistical inference across disciplines. The related optimization problems, in particular the maximization…
The concept of entropy, firstly introduced in information theory, rapidly became popular in many applied sciences via Shannon's formula to measure the degree of heterogeneity among observations. A rather recent research field aims at…
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to…
Information plays an important role in our understanding of the physical world. We hence propose an entropic measure of information for any physical theory that admits systems, states and measurements. In the quantum and classical world,…
In this paper, we address a longstanding challenge in self-organized criticality (SOC) systems: establishing a connection between sandpiles and complex networks. Our approach employs a similarity-based transfer function characterized by two…
Density-ratio estimation via classification is a cornerstone of unsupervised learning. It has provided the foundation for state-of-the-art methods in representation learning and generative modelling, with the number of use-cases continuing…
Most Neural Networks (NNs) for classification are trained using Cross-Entropy as a loss function. This approach requires the model to have an explicit classification layer. However, there exist alternative approaches, such as Contrastive…
This paper proposes a novel entropy encoding technique for lossless data compression. Representing a message string by its lexicographic index in the permutations of its symbols results in a compressed version matching Shannon entropy of…
Relative entropy is the standard measure of distinguishability in classical and quantum information theory. In the classical case, its loss under channels admits an exact chain rule, while in the quantum case only asymptotic, regularized…
We investigate the entanglement properties of the Quantum Six-Vertex Model on a cylinder, focusing on the Shannon-Renyi entropy in the limit of Renyi order $n = \infty$. This entropy, calculated from the ground state amplitudes of the…
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our…
We study a quantity called discrete layered entropy, which approximates the Shannon entropy within a logarithmic gap. Compared to the Shannon entropy, the discrete layered entropy is piecewise linear, approximates the expected length of the…
Traditional art pricing models often lack fine measurements of painting content. This paper proposes a new content measurement: the Shannon information quantity measured by the singular value decomposition (SVD) entropy of the painting…
Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics and data compression. Over…