Related papers: Entropy Estimation via Uniformization
Scientific and engineering processes deliver massive high-dimensional data sets that are generated as non-linear transformations of an initial state and few process parameters. Mapping such data to a low-dimensional manifold facilitates…
Entropy is useful in statistical problems as a measure of irreversibility, randomness, mixing, dispersion, and number of microstates. However, there remains ambiguity over the precise mathematical formulation of entropy, generalized beyond…
Entropy has emerged as a dynamic, interdisciplinary, and widely accepted quantitative measure of uncertainty across different disciplines. A unified understanding of entropy measures, supported by a detailed review of their theoretical…
We compare an entropy estimator $H_z$ recently discussed in [10] with two estimators $H_1$ and $H_2$ introduced in [6][7]. We prove the identity $H_z \equiv H_1$, which has not been taken into account in [10]. Then, we prove that the…
Recent work of Acharya et al. (NeurIPS 2019) showed how to estimate the entropy of a distribution $\mathcal D$ over an alphabet of size $k$ up to $\pm\epsilon$ additive error by streaming over $(k/\epsilon^3) \cdot…
Transfer entropy measures directed information flow in time series, and it has become a fundamental quantity in applications spanning neuroscience, finance, and complex systems analysis. However, existing estimation methods suffer from the…
Transfer entropy (TE) is an information theoretic measure that reveals the directional flow of information between processes, providing valuable insights for a wide range of real-world applications. This work proposes Transfer Entropy…
Entropy-type integral functionals of densities are widely used in mathematical statistics, information theory, and computer science. Examples include measures of closeness between distributions (e.g., density power divergence) and…
Complexity measures are essential to understand complex systems and there are numerous definitions to analyze one-dimensional data. However, extensions of these approaches to two or higher-dimensional data, such as images, are much less…
One of the fundamental problems in machine learning is the estimation of a probability distribution from data. Many techniques have been proposed to study the structure of data, most often building around the assumption that observations…
Efficient and accurate estimation of multivariate empirical probability distributions is fundamental to the calculation of information-theoretic measures such as mutual information and transfer entropy. Common techniques include variations…
To efficiently evaluate system reliability based on Monte Carlo simulation, importance sampling is used widely. The optimal importance sampling density was derived in 1950s for the deterministic simulation model, which maps an input to an…
This paper studies the problem of estimating the differential entropy $h(S+Z)$, where $S$ and $Z$ are independent $d$-dimensional random variables with $Z\sim\mathcal{N}(0,\sigma^2 \mathrm{I}_d)$. The distribution of $S$ is unknown, but $n$…
We present a method for computing the topological entropy of one-dimensional maps. As an approximation scheme, the algorithm converges rapidly and provides both upper and lower bounds.
We describe an approach to improving model fitting and model generalization that considers the entropy of distributions of modelling residuals. We use simple simulations to demonstrate the observational signatures of overfitting on ordered…
Balanced and efficient information flow is essential for optimizing language generation models. In this work, we propose Entropy-UID, a new token selection method that balances entropy and Uniform Information Density (UID) principles for…
A central task in analyzing complex dynamics is to determine the loci of information storage and the communication topology of information flows within a system. Over the last decade and a half, diagnostics for the latter have come to be…
The problem of assigning probability distributions which objectively reflect the prior information available about experiments is one of the major stumbling blocks in the use of Bayesian methods of data analysis. In this paper the method of…
Transfer entropy is capable of capturing nonlinear source-destination relations between multi-variate time series. It is a measure of association between source data that are transformed into destination data via a set of linear…
It is well known that to estimate the Shannon entropy for symbolic sequences accurately requires a large number of samples. When some aspects of the data are known it is plausible to attempt to use this to more efficiently compute entropy.…