Related papers: Minimax entropy: The statistical physics of optima…
The method of optimizing entropy is used to (i) conduct Asymptotic Hypothesis Testing and (ii) determine the particle distribution for which Entropy is maximized. This paper focuses on two related applications of Information Theory:…
The field of complex networks studies a wide variety of interacting systems by representing them as networks. To understand their properties and mutual relations, the randomisation of network connections is a commonly used tool. However,…
A maximum entropy-based framework is presented for the synthesis of projections from multiple Earth climate models. This identifies the most representative (most probable) model from a set of climate models -- as defined by specified…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible while constrained to match empirically estimated feature expectations. However, in many real-world…
We recount recent history behind building compact models of nonlinear, complex processes and identifying their relevant macroscopic patterns or "macrostates". We give a synopsis of computational mechanics, predictive rate-distortion theory,…
In the Minimum Description Length (MDL) principle, learning from the data is equivalent to an optimal coding problem. We show that the codes that achieve optimal compression in MDL are critical in a very precise sense. First, when they are…
In many real world problems, optimization decisions have to be made with limited information. The decision maker may have no a priori or posteriori data about the often nonconvex objective function except from on a limited number of points…
We review with a tutorial scope the information theory foundations of quantum statistical physics. Only a small proportion of the variables that characterize a system at the microscopic scale can be controlled, for both practical and…
Maximum entropy models provide the least constrained probability distributions that reproduce statistical properties of experimental datasets. In this work we characterize the learning dynamics that maximizes the log-likelihood in the case…
Properties of networks are often characterized in terms of features such as node degree distributions, average path lengths, diameters, or clustering coefficients. Here, we study shortest path length distributions. On the one hand, average…
Maximum entropy principle identifies forces conjugated to observables and the thermodynamic relations between them, independent upon their underlying mechanistic details. For data about state distributions or transition statistics, the…
We show that maximum entropy (maxent) models can be modeled with certain kinds of HMMs, allowing us to construct maxent models with hidden variables, hidden state sequences, or other characteristics. The models can be trained using the…
We apply the Principle of Maximum Entropy to the study of a general class of deterministic fractal sets. The scaling laws peculiar to these objects are accounted for by means of a constraint concerning the average content of information in…
Linear models are used in online decision making, such as in machine learning, policy algorithms, and experimentation platforms. Many engineering systems that use linear models achieve computational efficiency through distributed systems…
Maximum-entropy ensembles are key primitives in statistical mechanics from which thermodynamic properties can be derived. Over the decades, several approaches have been put forward in order to justify from minimal assumptions the use of…
We consider fitting a bivariate spline regression model to data using a weighted least-squares cost function, with weights that sum to one to form a discrete probability distribution. By applying the principle of maximum entropy, the weight…
Depending on context, the term entropy is used for a thermodynamic quantity, a~measure of available choice, a quantity to measure information, or, in the context of statistical inference, a maximum configuration predictor. For systems in…
Model compression is generally performed by using quantization, low-rank approximation or pruning, for which various algorithms have been researched in recent years. One fundamental question is: what types of compression work better for a…
Neural compression is the application of neural networks and other machine learning methods to data compression. Recent advances in statistical machine learning have opened up new possibilities for data compression, allowing compression…
Maximum Entropy is a powerful concept that entails a sharp separation between relevant and irrelevant variables. It is typically invoked in inference, once an assumption is made on what the relevant variables are, in order to estimate a…