Related papers: Relative entropy convergence under Picard's iterat…
Information theory on a time-discrete setting in the framework of time series analysis is generalized to the time-continuous case. Considerations of the Roessler and Lorenz dynamics as well as the Ornstein-Uhlenbeck process yield for…
Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables-which are neither discrete nor continuous-has not been available so far. Here, we present such an extension…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
It is a well-established fact in the scientific literature that Picard iterations of backward stochastic differential equations with globally Lipschitz continuous nonlinearity converge at least exponentially fast to the solution. In this…
This paper investigates achievable information rates and error exponents of mismatched decoding when the channel belongs to the class of channels that are close to the decoding metric in terms of relative entropy. For both discrete- and…
We study the convergence of a random iterative sequence of a family of operators on infinite dimensional Hilbert spaces, inspired by the Stochastic Gradient Descent (SGD) algorithm in the case of the noiseless regression, as studied in [1].…
Finding the minimal relative entropy of two quantum states under semidefinite constraints is a pivotal problem located at the mathematical core of various applications in quantum information theory. An efficient method for providing…
We propose an iterative estimating equations procedure for analysis of longitudinal data. We show that, under very mild conditions, the probability that the procedure converges at an exponential rate tends to one as the sample size…
The strong convergence of Euler approximations of stochastic delay differential equations is proved under general conditions. The assumptions on drift and diffusion coefficients have been relaxed to include polynomial growth and only…
Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics and data compression. Over…
We develop information-theoretic measures of spatial structure and pattern in more than one dimension. As is well known, the entropy density of a two-dimensional configuration can be efficiently and accurately estimated via a converging…
There is a vast body of recent literature on the reliability of communication through noisy channels, the recovery of community structures in the stochastic block model, the limiting behavior of the free entropy in spin glasses and the…
The specific relative entropy, introduced by N. Gantert, allows to quantify the discrepancy between the laws of potentially mutually singular measures. It appears naturally as the large deviations rate function in a randomized version of…
Relative entropy is a fundamental class of distances between probability distributions, with widespread applications in probability theory, statistics, and machine learning. In this work, we study relative entropy from a categorical…
Obtaining meaningful quantitative descriptions of the statistical dependence within multivariate systems is a difficult open problem. Recently, the Partial Information Decomposition (PID) was proposed to decompose mutual information (MI)…
New upper bounds on the relative entropy are derived as a function of the total variation distance. One bound refines an inequality by Verd\'{u} for general probability measures. A second bound improves the tightness of an inequality by…
This article shows how to combine the relative entropy method by D. Bresch, P.-E. Jabin, and Z. Wang in arXiv:1706.09564, arXiv:1906.04093 and the regularized $L^2(\mathbb{R}^d)$-estimate by Oelschl\"ager (Probability theory and related…
For discrete-time stochastic processes, there is a close connection between return/waiting times and entropy. Such a connection cannot be straightforwardly extended to the continuous-time setting. Contrarily to the discrete-time case one…
We study the convergence rates of the semi-discrete (SD) method originally proposed in Halidias (2012), Semi-discrete approximations for stochastic differential equations and applications, International Journal of Computer Mathematics,…
We consider the generalized differential entropy of normalized sums of independent and identically distributed (IID) continuous random variables. We prove that the R\'{e}nyi entropy and Tsallis entropy of order $\alpha\ (\alpha>0)$ of the…