Related papers: Information loss from dimensionality reduction in …
We analyze the geometry of a joint distribution over a set of discrete random variables. We briefly review Shannon's entropy, conditional entropy, mutual information and conditional mutual information. We review the entropic information…
A wide range of systems exhibit high dimensional incomplete data. Accurate estimation of the missing data is often desired, and is crucial for many downstream analyses. Many state-of-the-art recovery methods involve supervised learning…
We introduce a new protocol for a lossy data compression algorithm which is based on constraint satisfaction gates. We show that the theoretical capacity of algorithms built from standard parity-check gates converges exponentially fast to…
In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be…
The concept of Shannon entropy of random variables was generalized to measurable functions in general, and to simple functions with finite values in particular. It is shown that the information measure of a function is related to the time…
We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets.…
In the context of statistical learning, the Information Bottleneck method seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description…
We use phase space distributions specifically, the Wigner distribution (WD) and Husimi distribution (HD) to investigate certain information-theoretic measures as descriptors for a given system. We extensively investigate and analyze…
In this work, conditional entropy is used to quantify the information loss induced by passing a continuous random variable through a memoryless nonlinear input-output system. We derive an expression for the information loss depending on the…
Stochastic gradient descent (SGD) is a popular algorithm for optimization problems arising in high-dimensional inference tasks. Here one produces an estimator of an unknown parameter from independent samples of data by iteratively…
We consider Shannon entropy, Fisher information, R\'enyi entropy, and Tsallis entropy to study the quantum droplet phase in Bose-Einstein condensates. In the beyond mean-field description, the Gross-Pitaevskii equation with Lee-Huang-Yang…
Fabrication process variations are a major source of yield degradation in the nano-scale design of integrated circuits (IC), microelectromechanical systems (MEMS) and photonic circuits. Stochastic spectral methods are a promising technique…
The understanding of nonlinear, high dimensional flows, e.g, atmospheric and ocean flows, is critical to address the impacts of global climate change. Data Assimilation techniques combine physical models and observational data, often in a…
We provide a stochastic extension of the Baez-Fritz-Leinster characterization of the Shannon information loss associated with a measure-preserving function. This recovers the conditional entropy and a closely related information-theoretic…
The Shannon based conditional entropy that underlies five-dimensional Einstein-Hilbert gravity coupled to a dilaton field is investigated in the context of dynamical holographic AdS/QCD models. Considering the UV and IR dominance limits of…
High-dimensional imaging is becoming increasingly relevant in many fields from astronomy and cultural heritage to systems biology. Visual exploration of such high-dimensional data is commonly facilitated by dimensionality reduction.…
In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual…
Understanding geometric properties of natural language processing models' latent spaces allows the manipulation of these properties for improved performance on downstream tasks. One such property is the amount of data spread in a model's…
t-SNE has gained popularity as a dimension reduction technique, especially for visualizing data. It is well-known that all dimension reduction techniques may lose important features of the data. We provide a mathematical framework for…
In this dissertation we propose alternative analysis of distributed stochastic gradient descent (SGD) algorithms that rely on spectral properties of the data covariance. As a consequence we can relate questions pertaining to speedups and…