Related papers: Information-Theoretic Constraints on Variational Q…
As model sizes in machine learning continue to scale, distributed training is necessary to accommodate model weights within each device and to reduce training time. However, this comes with the expense of increased communication overhead…
The deterministic notions of capacity and entropy are studied in the context of communication and storage of information using square-integrable, bandlimited signals subject to perturbation. The $(\epsilon,\delta)$-capacity, that extends…
Variational quantum computing offers a flexible computational paradigm with applications in diverse areas. However, a key obstacle to realizing their potential is the Barren Plateau (BP) phenomenon. When a model exhibits a BP, its parameter…
Thermodynamics and information have intricate interrelations. Often thermodynamics is considered to be the logical premise to justify that information is physical - through Landauer's principle -, thereby also linking information and…
The variational quantum algorithm (VQA) with a parametrized quantum circuit is widely applicable to near-term quantum computing, but its fundamental issues that limit optimization performance have been reported in the literature. For…
Throughout the cognitive-science literature, there is widespread agreement that decision-making agents operating in the real world do so under limited information-processing capabilities and without access to unbounded cognitive or…
This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate…
The training dynamics of hidden layers in deep learning are poorly understood in theory. Recently, the Information Plane (IP) was proposed to analyze them, which is based on the information-theoretic concept of mutual information (MI). The…
Information based thermodynamic logic is revisited. It consists of two parts: Part A applies the modern theory of probability in which an arbitrary convex function \phi is employed as an analytic "device" to express information as…
Data compression can be achieved by reducing the dimensionality of high-dimensional but approximately low-rank datasets, which may in fact be described by the variation of a much smaller number of parameters. It often serves as a…
The prospect of achieving quantum advantage with Quantum Neural Networks (QNNs) is exciting. Understanding how QNN properties (e.g., the number of parameters $M$) affect the loss landscape is crucial to the design of scalable QNN…
We introduce a constructive framework for assigning thermodynamic structure to an arbitrary data system from its measured microstates. Starting from an empirical distribution over configurations, we first infer a data-driven energy function…
Learning with hidden variables is a central challenge in probabilistic graphical models that has important implications for many real-life problems. The classical approach is using the Expectation Maximization (EM) algorithm. This…
This paper mainly discusses the diffusion on complex networks with time-varying couplings. We propose a model to describe the adaptive diffusion process of local topological and dynamical information, and find that the Barabasi-Albert…
Quantum information scrambling refers to the loss of local recoverability of quantum information, which has found widespread attention from high energy physics to quantum computing. In the present analysis we propose a possible starting…
We address the question of characterizing and finding optimal representations for supervised learning. Traditionally, this question has been tackled using the Information Bottleneck, which compresses the inputs while retaining information…
Quantum Darwinism explains the emergence of classical objectivity through the redundant encoding of pointer information in environmental fragments. However, existing diagnostics rely on arbitrary thresholds or structural assumptions that…
Although large language models rely on chain-of-thought for complex reasoning, the overthinking phenomenon severely degrades inference efficiency. Existing reinforcement learning methods compress reasoning chains by designing elaborate…
This paper proposes a novel perspective on learning, positing it as the pursuit of dynamical invariants -- data combinations that remain constant or exhibit minimal change over time as a system evolves. This concept is underpinned by both…
Adaptive physical and biological systems continually process fluctuating information from their environments. When the environment is nonstationary, inference itself becomes a nonequilibrium process with thermodynamic cost. We analyse a…