Related papers: Entropy landscape of solutions in the binary perce…
Understanding and controlling the informational complexity of neural networks is a central challenge in machine learning, with implications for generalization, optimization, and model capacity. While most approaches rely on entropy-based…
In this study, we devise a model that introduces two hierarchies into information entropy. The two hierarchies are the size of the region for which entropy is calculated and the size of the component that determines whether the structures…
This paper studies the interpretability of neural network features from a Bayesian Gaussian view, where optimizing a cost is reaching a probabilistic bound; learning a model approximates a density that makes the bound tight and the cost…
Like natural complex systems such as the Earth's climate or a living cell, semiconductor lithography systems are characterized by nonlinear dynamics across more than a dozen orders of magnitude in space and time. Thousands of sensors…
To comprehend the hierarchical organization of large integrated systems, we introduce the hierarchical map equation, which reveals multilevel structures in networks. In this information-theoretic approach, we exploit the duality between…
A perceptron is trained by a random bit sequence. In comparison to the corresponding classification problem, the storage capacity decreases to alpha_c=1.70\pm 0.02 due to correlations between input and output bits. The numerical results are…
LiDAR point cloud compression is vital for autonomous systems to handle massive data from high-resolution sensors. While learned entropy modeling built upon octree structures yields high compression gains, it faces two critical bottlenecks:…
This work presents a novel means for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel, specifically entropy and trace, yield…
We investigate the VC-dimension of the perceptron and simple two-layer networks like the committee- and the parity-machine with weights restricted to values $\pm1$. For binary inputs, the VC-dimension is determined by atypical pattern sets,…
Recent work has shown that probabilistic models based on pairwise interactions-in the simplest case, the Ising model-provide surprisingly accurate descriptions of experiments on real biological networks ranging from neurons to genes.…
Spatial road networks have been widely employed to model the structure and connectivity of cities. In such representation, the question of spatial scale of the entities in the network, i.e. what its nodes and edges actually embody in…
We study the ground state entanglement entropy of the quantum Dyson hierarchical spin chain in which the interaction decays algebraically with the distance as $r^{-1-\sigma}$. We exploit the real-space renormalisation group solution which…
Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics and data compression. Over…
The landscape of low-energy effective field theories stemming from string theory is too vast for a systematic exploration. However, the meadows of the string landscape may be fertile ground for the application of machine learning…
It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete,…
In order to efficiently explore the chemical space of all possible small molecules, a common approach is to compress the dimension of the system to facilitate downstream machine learning tasks. Towards this end, we present a data driven…
Recently, deep learning models have achieved great success in computer vision applications, relying on large-scale class-balanced datasets. However, imbalanced class distributions still limit the wide applicability of these models due to…
Understanding how species persist under interacting stressors is a central challenge in ecology. We develop a spatially explicit reaction-diffusion framework to investigate competing species in landscapes shaped by climate variability,…
Entropy relates the fast, microscopic behaviour of the elements in a system to its slow, macroscopic state. We propose to use it to explain how, as complexity theory suggests, small scale decisions of individuals form cities. For this, we…
Semi-supervised learning has emerged as an appealing strategy to train deep models with limited supervision. Most prior literature under this learning paradigm resorts to dual-based architectures, typically composed of a teacher-student…