Related papers: Computationally Efficient Approximations for Matri…

We present a new type of generalization of the Renyi entropy that follows naturally from its representation as a thermodynamic quantity. We apply it to the case of d-dimensional conformal field theories (CFTs) reduced on a region bounded by…

At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources. A common approach to reduce the cost of inference is to reduce its memory…

Let A be a matrix, c be any linear objective function and x be a fractional vector, say an LP solution to some discrete optimization problem. Then a recurring task in theoretical computer science (and in approximation algorithms in…

This paper introduces "swiveled Renyi entropies" as an alternative to the Renyi entropic quantities put forward in [Berta et al., Phys. Rev. A 91, 022333 (2015)]. What distinguishes the swiveled Renyi entropies from the prior proposal of…

Existing theories on deep nonparametric regression have shown that when the input data lie on a low-dimensional manifold, deep neural networks can adapt to the intrinsic data structures. In real world applications, such an assumption of…

To accelerate kernel methods, we propose a near input sparsity time algorithm for sampling the high-dimensional feature space implicitly defined by a kernel transformation. Our main contribution is an importance sampling method for…

Given a large real symmetric, positive semidefinite m-by-m matrix, the goal of this paper is to show how a numerical approximation of the entropy, given by the sum of the entropies of the individual eigenvalues, can be computed in an…

We present simple and computationally efficient nonparametric estimators of R\'enyi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\R^d$. The estimators are…

Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability. However, it is by no means obvious how to estimate…

Information has an entropic character which can be analyzed within the Statistical Theory in molecular systems. R. Landauer and C.H. Bennett showed that a logical copy can be carried out in the limit of no dissipation if the computation is…

We set out to demonstrate that the R\'enyi entropies with parameter $\alpha$ are better thought of as operating in a type of non-linear semiring called a positive semifield. We show how the R\'enyi's postulates lead to Pap's g-calculus…

Information theoretic quantities play a central role in machine learning. The recent surge in the complexity of data and models has increased the demand for accurate estimation of these quantities. However, as the dimension grows the…

Shannon entropy is widely used to measure the complexity of DNA sequences but suffers from saturation effects that limit its discriminative power for long uniform segments. We introduce a novel metric, the entropy rank ratio R, which…

Shannon and Renyi entropies are quantitative measures of uncertainty in a data set. They are developed by Renyi in the context of entropy theory. These measures have been studied in the case of the multivariate t-distributions. We extend…

Entropy and relative or cross entropy measures are two very fundamental concepts in information theory and are also widely used for statistical inference across disciplines. The related optimization problems, in particular the maximization…

Approximation of entropies of various types using machine learning (ML) regression methods are shown for the first time. The ML models presented in this study define the complexity of the short time series by approximating dissimilar…

We extend the approach of Casini, Huerta and Myers to a new calculation of the Renyi entropy of a general CFT in d dimensions with a spherical entangling surface, in terms of certain thermal partition functions. We apply this approach to…

Entropy is a fundamental concept in equilibrium statistical mechanics, yet its origin in the non-equilibrium dynamics of isolated quantum systems is not fully understood. A strong consensus is emerging around the idea that the stationary…

We consider the problem of approximating the von Neumann entropy of a large, sparse, symmetric positive semidefinite matrix $A$, defined as $\operatorname{tr}(f(A))$ where $f(x)=-x\log x$. After establishing some useful properties of this…

The measured relative entropy and measured R\'enyi relative entropy are quantifiers of the distinguishability of two quantum states $\rho$ and $\sigma$. They are defined as the maximum classical relative entropy or R\'enyi relative entropy…