Related papers: The Entropy Gain of Linear Time-Invariant Filters …
In [1] it is shown that recurrent neural networks (RNNs) can learn - in a metric entropy optimal manner - discrete time, linear time-invariant (LTI) systems. This is effected by comparing the number of bits needed to encode the…
One of the most influential results in neural network theory is the universal approximation theorem [1, 2, 3] which states that continuous functions can be approximated to within arbitrary accuracy by single-hidden-layer feedforward neural…
Despite growing interest in data stream mining the most successful incremental learners, such as VFDT, still use periodic recomputation to update attribute information gains and Gini indices. This note provides simple incremental formulas…
Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and…
This article examines large time behaviour of finite state mean-field interacting particle systems. Our first main result is a sharp estimate (in the exponential scale) on the time required for convergence of the empirical measure process…
Increasing the number $N$ of elements of a system typically makes the entropy to increase. The question arises on {\it what particular entropic form} we have in mind and {\it how it increases} with $N$. Thermodynamically speaking it makes…
This work approaches the problem of computing incremental $\ell_1$ and $\ell_\infty$ gains for discrete-time positive systems in \lure feedback with static memoryless nonlinearities, and regulating the $\ell_\infty$ gain through the design…
Entropy rate of sequential data-streams naturally quantifies the complexity of the generative process. Thus entropy rate fluctuations could be used as a tool to recognize dynamical perturbations in signal sources, and could potentially be…
The log-periodic equation for the entropy $S = - (k/a) \sum_{i=1}^{N} p_{i} \sin(a \ln p_{i})$, based on the forgotten Sharma-Taneja entropy measure, is studied for the first time with $N$ the total number of system states and $p_{i}$ the…
The study considers advantages of the introduced measure of time based on the entropy change under irreversible processes (entropy production). Using the example of non-equilibrium expansion of an ideal gas in vacuum, such a measure is…
The time variation of entropy, as an alternative to the variance, is proposed as a measure of the diffusion rate. It is shown that for linear and time-translationally invariant systems having a large-time limit for the density, at large…
In this paper, the linear Gaussian relay problem is considered. Under the linear time-invariant (LTI) model the problem is formulated in the frequency domain based on the Toeplitz distribution theorem. Under the further assumption of…
On a fine grained scale the Gibbs entropy of an isolated system remains constant throughout its dynamical evolution. This is a consequence of Liouville's theorem for Hamiltonian systems and appears to contradict the second law of…
We study the unitary time evolution of the entropy of entanglement of a one-dimensional system between the degrees of freedom in an interval of length l and its complement, starting from a pure state which is not an eigenstate of the…
We extend the notion of estimation entropy of autonomous dynamical systems proposed by Liberzon and Mitra [1] to nonlinear dynamical systems with uncertain inputs with bounded variation. We call this new notion the {$\epsilon$}-estimation…
A quantity of interest to characterise continuous-valued stochastic processes is the differential entropy rate. The rate of convergence of many properties of LRD processes is slower than might be expected, based on the intuition for…
Recursive filters are treated as linear time-invariant (LTI) systems but they are not: uninitialised, they have an infinite number of outputs for any given input, while if initialised, they are not time-invariant. This short tutorial…
The aim of this paper is to shed light on the analysis of non-stationary time series by means of the method of diffusion entropy. For this purpose, we first study the case when infinitely many time series, as different realizations of the…
The Geometrically Intrinsic Nonlinear Recursive Filter, or GI Filter, is designed to estimate an arbitrary continuous-time Markov diffusion process X subject to nonlinear discrete-time observations. The GI Filter is fundamentally different…
Entropy comparison inequalities are obtained for the differential entropy $h(X+Y)$ of the sum of two independent random vectors $X,Y$, when one is replaced by a Gaussian. For identically distributed random vectors $X,Y$, these are closely…