Related papers: Entropy, Information, and the Updating of Probabil…
We propose a novel estimator of the mutual information between two ordinal vectors $x$ and $y$. Our approach is inductive (as opposed to deductive) in that it depends on the data generating distribution solely through some nonparametric…
We advocate the use of a notion of entropy that reflects the relative abundances of the symbols in an alphabet, as well as the similarities between them. This concept was originally introduced in theoretical ecology to study the diversity…
A central challenge in statistical inference is the presence of confounding variables that may distort observed associations between treatment and outcome. Conventional "causal" methods, grounded in assumptions such as ignorability, exclude…
Herding is a deterministic algorithm used to generate data points that can be regarded as random samples satisfying input moment conditions. The algorithm is based on the complex behavior of a high-dimensional dynamical system and is…
For the purpose of causal inference we employ a stochastic model of the data generating process, utilizing individual propensity probabilities for the treatment, and also individual and counterfactual prognosis probabilities for the…
The asymptotic convergence of probability density function (pdf) and convergence of differential entropy are examined for the non-stationary processes that follow the maximum entropy principle (MaxEnt) and maximum entropy production…
Maximization of an expensive, unimodal function under random observations has been an important problem in hyperparameter tuning. It features expensive function evaluations (which means small budgets) and a high level of noise. We develop…
In this review, we examine an extended Bayesian inference method and its relation to biological information processing. We discuss the idea of combining two modes of Bayesian inference. The first is the standard Bayesian inference, which…
The goal of this paper is to explore the basic Approximate Bayesian Computation (ABC) algorithm via the lens of information theory. ABC is a widely used algorithm in cases where the likelihood of the data is hard to work with or…
We present arguments for the formulation of unified approach to different standard continuous inference methods from partial information. It is claimed that an explicit partition of information into a priori (prior knowledge) and a…
Traditionally, the MaxEnt workshops start by a tutorial day. This paper summarizes my talk during 2001'th workshop at John Hopkins University. The main idea in this talk is to show how the Bayesian inference can naturally give us all the…
Stability selection is a versatile framework for structure estimation and variable selection in high-dimensional setting, primarily grounded in frequentist principles. In this paper, we propose an enhanced methodology that integrates…
The field of complex networks studies a wide variety of interacting systems by representing them as networks. To understand their properties and mutual relations, the randomisation of network connections is a commonly used tool. However,…
Inference and decision making under uncertainty are key processes in every autonomous system and numerous robotic problems. In recent years, the similarities between inference and decision making triggered much work, from developing unified…
Estimation of permutation entropy (PE) using Bayesian statistical methods is presented for systems where the ordinal pattern sampling follows an independent, multinomial distribution. It is demonstrated that the PE posterior distribution is…
Various properties of relative entropy have led to its widespread use in information theory. These properties suggest that relative entropy has a role to play in systems that attempt to perform inference in terms of probability…
We prove that information-theoretic maximum entropy (MaxEnt) approach to canonical ensemble is mathematically equivalent to the classic approach of Boltzmann, Gibbs and Darwin-Fowler. The two approaches, however, "interpret" a same…
We explore the use of the method of Maximum Entropy (ME) as a technique to generate approximations. In a first use of the ME method the "exact" canonical probability distribution of a fluid is approximated by that of a fluid of hard…
We can perform inference in Bayesian belief networks by enumerating instantiations with high probability thus approximating the marginals. In this paper, we present a method for determining the fraction of instantiations that has to be…
An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output…