Related papers: Measures of Entropy from Data Using Infinitely Div…
This paper introduces a novel approach to probabilistic deep learning, kernel density matrices, which provide a simpler yet effective mechanism for representing joint probability distributions of both continuous and discrete random…
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the…
Indeterminacy associated with probing of a quantum state is commonly expressed through spectral distances (metric) featured in the outcomes of repeated experiments. Here we express it as an effective amount (measure) of distinct outcomes…
Entropy estimation is of practical importance in information theory and statistical science. Many existing entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for…
Nonparametric density estimation for compositional data supported on the simplex is examined under a missing at random mechanism. Rather than imputing missing values and estimating the density from a completed data set, we adopt a strategy…
Observational entropy provides a general notion of quantum entropy that appropriately interpolates between Boltzmann's and Gibbs' entropies, and has recently been argued to provide a useful measure of out-of-equilibrium thermodynamic…
We introduce a constructive framework for assigning thermodynamic structure to an arbitrary data system from its measured microstates. Starting from an empirical distribution over configurations, we first infer a data-driven energy function…
Gaussian processes (GPs) are ubiquitous tools for modeling and predicting continuous processes in physical and engineering sciences. This is partly due to the fact that one may employ a Gaussian process as an interpolator while facilitating…
In this paper we investigate the problem of estimating the regression function in models with correlated observations. The data is obtained from several experimental units each of them forms a time series. We propose a new estimator based…
This paper applies the recently axiomatized Optimum Information Principle (minimize the Kullback-Leibler information subject to all relevant information) to nonparametric density estimation, which provides a theoretical foundation as well…
Data transformation, e.g. feature transformation and selection, is an integral part of any machine learning procedure. In this paper we introduce an information-theoretic model and tools to assess the quality of data transformations in…
Some general considerations on the notion of entropy in physics are presented. An attempt is made to clarify the question of the differentiation between physical entropy (the Clausius-Boltzmann one) and quantities called entropies…
We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was…
Comparing probability distributions is a core challenge across the natural, social, and computational sciences. Existing methods, such as Maximum Mean Discrepancy (MMD), struggle in high-dimensional and non-compact domains. Here we…
We investigate fundamental connections between thermodynamics and quantum information theory. First, we show that the operational framework of thermal operations is nonequivalent to the framework of Gibbs-preserving maps, and we comment on…
We present simple and computationally efficient nonparametric estimators of R\'enyi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\R^d$. The estimators are…
These lectures deal with the problem of inductive inference, that is, the problem of reasoning under conditions of incomplete information. Is there a general method for handling uncertainty? Or, at least, are there rules that could in…
Thermodynamic entropy, as defined by Clausius, characterizes macroscopic observations of a system based on phenomenological quantities such as temperature and heat. In contrast, information-theoretic entropy, introduced by Shannon, is a…
Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics and data compression. Over…
Tactical selection of experiments to estimate an underlying model is an innate task across various fields. Since each experiment has costs associated with it, selecting statistically significant experiments becomes necessary. Classic linear…