Related papers: Three tutorial lectures on entropy and counting
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a…
Entropy is the distinguishing and most important concept of our efforts to understand and regularize our observations of a very large class of natural phenomena, and yet, it is one of the most contentious concepts of physics. In this…
The profile of a sample is the multiset of its symbol frequencies. We show that for samples of discrete distributions, profile entropy is a fundamental measure unifying the concepts of estimation, inference, and compression. Specifically,…
Transfer entropy is used to establish a measure of causal relationships between two variables. Symbolic transfer entropy, as an estimation method for transfer entropy, is widely applied due to its robustness against non-stationarity. This…
These notes are meant to provide a rapid introduction to triangulated categories. We start with the definition of an additive category and end with a glimps of tilting theory. Some exercises are included.
We give a notion of entropy for general gemetric structures, which generalizes well-known notions of topological entropy of vector fields and geometric entropy of foliations, and which can also be applied to singular objects, e.g. singular…
The entropy of a hierarchical network topology in an ensemble of sparse random networks with "hidden variables" associated to its nodes, is the log-likelihood that a given network topology is present in the chosen ensemble.We obtain a…
Computational entropies provide a framework for quantifying uncertainty and randomness under computational constraints. They play a central role in classical cryptography, underpinning the analysis and construction of primitives such as…
We prove existence of asymptotic entropy of random walks on regular languages over a finite alphabet and we give formulas for it. Furthermore, we show that the entropy varies real-analytically in terms of probability measures of constant…
The phenomenon of entropy concentration provides strong support for the maximum entropy method, MaxEnt, for inferring a probability vector from information in the form of constraints. Here we extend this phenomenon, in a discrete setting,…
A new combinatorial-probabilistic diagnostic entropy has been introduced. It describes the pair-wise sum of probabilities of system conditions that have to be distinguished during the diagnosing process. The proposed measure describes the…
It is shown for classical and quantum ensembles that there is a unique quantity which has the properties of a "volume". This quantity is a function of the ensemble entropy, and hence provides a geometric interpretation for the latter. It…
We illustrate the use of the notion of derived recurrences introduced earlier to evaluate the algebraic entropy of self-maps of projective spaces. We in particular give an example, where a complete proof is still awaited, but where…
The paper is devoted to discretization of integral norms of functions from a given finite dimensional subspace. This problem is very important in applications but there is no systematic study of it. We present here a new technique, which…
We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find…
A definable set $X$ in the first-order language of rings defines a family of random vectors: for each finite field $\mathbb{F}_q$, let the distribution be supported and uniform on the $\mathbb{F}_q$-rational points of $X$. We employ results…
Notes for a Course on Probability and Statistics: L1: Elements of Probability; L2: Bayesian Inference; L3: Monte Carlo Methods
We overview the notion of entropy in thermodynamics. We start from the smooth case using differential forms on the manifold, which is the natural language for thermodynamics. Then the axiomatic definition of entropy as ordering on a set…
This is a series of lecture notes explaining topos theory and its application in physics.
We present the concept of the \emph{information efficiency of functions} as a technique to understand the interaction between information and computation. Based on these results we identify a new class of objects that we call…