Related papers: Innovation Capacity of Dynamical Learning Systems

Learning performed over finite time is inherently irreversible. In Part~I of this series, we modeled learning as a transport process in the space of parameter distributions and derived the Epistemic Speed Limit (ESL), which lower-bounds…

To explain conceptual gap between classical/quantum and other, hypothetical descriptions of world, several principles has been proposed. So far, all these principles have not explicitly included the uncertainty relation. Here we introduce…

Standard supervised learning optimizes for predictive accuracy but remains agnostic to the internal geometry of learned features, often yielding representations that are entangled and brittle. We propose Class-Conditional Activation…

A significant use case of instruction-finetuned Large Language Models (LLMs) is to solve question-answering tasks interactively. In this setting, an LLM agent is tasked with making a prediction by sequentially querying relevant information…

In our companion paper [1], an information identity decomposition has been derived, which can be interpreted as a law of conservation of information flows in feedback systems. In this paper, we further investigate this decomposition result…

Neural networks have gained importance as the machine learning models that achieve state-of-the-art performance on large-scale image classification, object detection and natural language processing tasks. In this paper, we consider noisy…

We introduce a dynamical model of node repair in distributed storage systems wherein the storage nodes are subjected to failures according to independent Poisson processes. The main parameter that we study is the time-average capacity of…

Quantum memory systems are vital in quantum information processing for dependable storage and retrieval of quantum states. Inspired by classical reliability theories that synthesize reliable computing systems from unreliable components, we…

Stochastic neural networks are a prototypical computational device able to build a probabilistic representation of an ensemble of external stimuli. Building on the relationship between inference and learning, we derive a synaptic plasticity…

We adopt an information-theoretic framework to analyze the generalization behavior of the class of iterative, noisy learning algorithms. This class is particularly suitable for study under information-theoretic metrics as the algorithms are…

Physical reservoir computing is a computational framework that offers an energy- and computation-efficient alternative to conventional training of neural networks. In reservoir computing, input signals are mapped into the high-dimensional…

Energy-based learning algorithms, such as predictive coding (PC), have garnered significant attention in the machine learning community due to their theoretical properties, such as local operations and biologically plausible mechanisms for…

The ability to store quantum information without recourse to constant feedback processes would yield a significant advantage for future implementations of quantum information processing. In this paper, limitations of the prototypical model,…

A persistent paradox in continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their output predictions fail. We formalize this distinction as the gap between deep…

In this paper, we consider some long-standing problems in communication systems with access to noisy feedback. We introduce a new notion, the residual directed information, to capture the effective information flow (i.e. mutual information…

Catastrophic forgetting is not an engineering failure. It is a mathematical consequence of storing knowledge as global parameter superposition. Existing methods, such as regularization, replay, and frozen subnetworks, add external…

Reservoir Computing (RC), a type of recurrent random neural network, is a powerful framework for modeling complex and chaotic dynamics. However, its autonomous (closed-loop) operation is often plagued by inherent instability. Moreover,…

We study the maximum achievable differential entropy at the output of a system assigning to each input X the sum X+N, with N a given noise with probability law absolutely continuous with respect to the Lebesgue measure and where the input…

This paper considers the subject of information losses arising from the finite datasets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated…

Typically, real-world stochastic processes are not easy to analyze. In this work we study the representation of any stochastic process as a memoryless innovation process triggering a dynamic system. We show that such a representation is…