Related papers: Information Loss in Static Nonlinearities
This work proposes a new loss function targeting classification problems, utilizing a source of information overlooked by cross entropy loss. First, we derive a series of the tightest upper and lower bounds for the probability of a random…
The Principle of Maximum Entropy is a rigorous technique for estimating an unknown distribution given partial information while simultaneously minimizing bias. However, an important requirement for applying the principle is that the…
This paper develops a data-driven framework for stabilization of discrete-time infinite-dimensional systems. We investigate informativity for stabilization, defined as the existence of a feedback gain that stabilizes all systems compatible…
We present a measurement noise reduction scheme based on information flow of a chaotic system. This scheme operates on conditions of chaoticity and well-defined noise level, not depending on other detailed characteristics of noise. Starting…
In this paper we develop the concept of information transfer between the Borel-measurable sets for a dynamical system described by a measurable space and a non-singular transformation. The concept is based on how Shannon entropy is…
What is information originating in observation? Until now it has no scientifically conclusive definition. Information is memorized entropy cutting in random observations which processing interactions. Randomness of various interactive…
We theoretically investigate how information flows when two particles interact with each other. Understanding the physical mechanisms of directional information flow is crucial for advancing information thermodynamics and stochastic…
Exchange of information between a quantum system and its surrounding environment plays a fundamental role in the study of the dynamics of open quantum systems. Here we discuss the role of the information exchange in the non-Markovian…
Measuring the average information that is necessary to describe the behaviour of a dynamical system leads to a generalization of the Kolmogorov-Sinai entropy. This is particularly interesting when the system has null entropy and the…
The major problem in information theoretic analysis of neural responses and other biological data is the reliable estimation of entropy--like quantities from small samples. We apply a recently introduced Bayesian entropy estimator to…
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
Entropy estimation, due in part to its connection with mutual information, has seen considerable use in the study of time series data including causality detection and information flow. In many cases, the entropy is estimated using…
This paper deals with developing tests for checking whether an unknown system has certain structural properties. The tests that we are aiming at are in terms of noisy input-state-output data obtained from the unknown system. Since, in…
We discuss the information entropy for a general open pointer-based simultaneous measurement and show how it is bound from below. This entropic uncertainty bound is a direct consequence of the structure of the entropy and can be obtained…
Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building…
The work [8] established memory loss in the time-dependent (non-random) case of uniformly expanding maps of the interval. Here we find conditions under which we have convergence to the normal distribution of the appropriately scaled…
We study nonstationary dynamical systems formed by sequential concatenation of nonuniformly expanding maps with a uniformly expanding first return map. Assuming a polynomially decaying upper bound on the tails of first return times that is…
Accurate information processing is crucial both in technology and in nature. To achieve it, any information processing system needs an initial supply of resources away from thermal equilibrium. Here we establish a fundamental limit on the…
The formalism of statistical mechanics can be generalized by starting from more general measures of information than the Shannon entropy and maximizing those subject to suitable constraints. We discuss some of the most important examples of…
Entropy is a central concept in physics, but can be challenging to calculate even for systems that are easily simulated. This is exacerbated out of equilibrium, where generally little is known about the distribution characterizing simulated…