Related papers: Entropy Estimation via Uniformization
Information theoretic quantities play a central role in machine learning. The recent surge in the complexity of data and models has increased the demand for accurate estimation of these quantities. However, as the dimension grows the…
The notion of metric entropy dimension is introduced to measure the complexity of entropy zero dynamical systems. For measure preserving systems, we define entropy dimension via the dimension of entropy generating sequences. This…
In a paper [8] the authors classify entropy into three categories, as a thermodynamics quantity, as a measure of information production, and as a means of statistical inference. An entropy measure introduced by Mathai falls into the second…
Entropy estimation plays a crucial role in various fields, such as information theory, statistical data science, and machine learning. However, traditional entropy estimation methods often struggle with complex data distributions.…
We propose utilizing entropy as a diagnostic tool to distinguish between constant and dynamical dark energy models. Entropy, a measure of the system's disorder or information content, captures the complexity and evolution of the universe.…
Finding better solutions to combinatorial optimization problems could have a large positive impact on many real-world application areas, such as logistics. For this reason, significant efforts have been made to design novel optimisation…
We propose a novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data to the target distribution, for which MI is easier to…
We describe a method to computationally estimate the probability density function of a univariate random variable by applying the maximum entropy principle with some local conditions given by Gaussian functions. The estimation errors and…
A brief discussion is given of the traditional version of the Maximum Entropy Method, including a review of some of the criticism that has been made in regard to its use in statistical inference. Motivated by these questions, a modified…
In many scientific applications, the target probability distribution cannot be evaluated in closed form or sampled from directly. Instead, it can often be decomposed into multiple components, some of which are accessible only through…
In this work, we demonstrate how to reliably estimate epistemic uncertainty while maintaining the flexibility needed to capture complicated aleatoric distributions. To this end, we propose an ensemble of Normalizing Flows (NF), which are…
Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data…
The method of maximum entropy is quite a powerful tool to solve the generalized moment problem, which consists of determining the probability density of a random variable X from the knowledge of the expected values of a few functions of the…
Estimating Mutual Information (MI), a key measure of dependence of random quantities without specific modelling assumptions, is a challenging problem in high dimensions. We propose a novel mutual information estimator based on parametrizing…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible while constrained to match empirically estimated feature expectations. However, in many real-world…
In inverse problems, the parameters of a model are estimated based on observations of the model response. The Bayesian approach is powerful for solving such problems; one formulates a prior distribution for the parameter state that is…
In this letter we propose the use of physics techniques for entropy determination on constrained parameter optimization problems. The main feature of such techniques, the construction of an unbiased walk on energy space, suggests their use…
We present a novel approach for test-time adaptation via online self-training, consisting of two components. First, we introduce a statistical framework that detects distribution shifts in the classifier's entropy values obtained on a…
Knowledge distillation (KD) enables a smaller "student" model to mimic a larger "teacher" model by transferring knowledge from the teacher's output or features. However, most KD methods treat all samples uniformly, overlooking the varying…
Partly motivated by entropy-estimation problems in neuroscience, we present a detailed and extensive comparison between some of the most popular and effective entropy estimation methods used in practice: The plug-in method, four different…