Related papers: Cross Entropy Approximation of Structured Covarian…
In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to…
This paper proposes a versatile covariate adjustment method that directly incorporates covariate balance in regression discontinuity (RD) designs. The new empirical entropy balancing method reweights the standard local polynomial RD…
This paper studies the problem of interacting multiple model (IMM) estimation for jump Markov linear systems with unknown measurement noise covariance. The system state and the unknown covariance are jointly estimated in the framework of…
This paper deals with the problem of estimating predictive densities of a matrix-variate normal distribution with known covariance matrix. Our main aim is to establish some Bayesian predictive densities related to matricial shrinkage…
Kernel techniques are among the most popular and flexible approaches in data science allowing to represent probability measures without loss of information under mild conditions. The resulting mapping called mean embedding gives rise to a…
Effective uncertainty quantification is important for training modern predictive models with limited data, enhancing both accuracy and robustness. While Bayesian methods are effective for this purpose, they can be challenging to scale. When…
Multivariate datasets are common in various real-world applications. Recently, copulas have received significant attention for modeling dependencies among random variables. A copula-based information measure is required to quantify the…
We examine the {combinatorial} or {probabilistic} definition ("Boltzmann's principle") of the entropy or cross-entropy function $H \propto \ln \mathbb{W}$ or $D \propto - \ln \mathbb{P}$, where $\mathbb{W}$ is the statistical weight and…
In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be…
Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and…
Rare event probability estimation is an important topic in reliability analysis. Stochastic methods, such as importance sampling, have been developed to estimate such probabilities but they often fail in high dimension. In this paper, we…
In this work, we consider the notion of "criterion collapse," in which optimization of one metric implies optimality in another, with a particular focus on conditions for collapse into error probability minimizers under a wide variety of…
We establish a connection between barcode entropy and metric entropy. Namely, we show that the barcode entropy bounds the metric entropy from below for a measure from a specific class of invariant measures associated with a pair of…
Model or variable selection is usually achieved through ranking models according to the increasing order of preference. One of methods is applying Kullback-Leibler distance or relative entropy as a selection criterion. Yet that will raise…
We introduce a Maximum Entropy model able to capture the statistics of melodies in music. The model can be used to generate new melodies that emulate the style of the musical corpus which was used to train it. Instead of using the $n-$body…
Deep learning systems have been reported to acheive state-of-the-art performances in many applications, and one of the keys for achieving this is the existence of well trained classifiers on benchmark datasets which can be used as backbone…
The second law of thermodynamics governs that nonequilibrium systems evolve towards states of higher entropy over time. However, it does not specify the rate of this evolution and the role of fluctuations that impact the system's dynamics.…
The use of correntropy as a similarity measure has been increasing in different scenarios due to the well-known ability to extract high-order statistic information from data. Recently, a new similarity measure between complex random…
We propose the Linearly Adaptive Cross Entropy Loss function. This is a novel measure derived from the information theory. In comparison to the standard cross entropy loss function, the proposed one has an additional term that depends on…
This paper applies the recently axiomatized Optimum Information Principle (minimize the Kullback-Leibler information subject to all relevant information) to nonparametric density estimation, which provides a theoretical foundation as well…