Related papers: Innovation Capacity of Dynamical Learning Systems
The cognitive interference channel (C-IFC) consists of a classical two-user interference channel in which the message of one user (the "primary" user) is non-causally available at the transmitter of the other user (the "cognitive" user). We…
This work presents a complete geometrical characterisation of divisible and indivisible time-evolution at the level of probabilities for systems with two configurations, open or closed. Our new geometrical construction in the space of…
Common information (CI) is ubiquitous in information theory and related areas such as theoretical computer science and discrete probability. However, because there are multiple notions of CI, a unified understanding of the deep…
We study the information transmission capacities of quantum Markov semigroups $(\Psi^t)_{t\in \mathbb{N}}$ acting on $d-$dimensional quantum systems. We show that, in the limit of $t\to \infty$, the capacities can be efficiently computed in…
Variational dimensionality reduction methods are widely used for their accuracy, generative capabilities, and robustness. We introduce a unifying framework that generalizes both such as traditional and state-of-the-art methods. The…
The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a…
The rising computational and energy demands of artificial intelligence systems urge the exploration of alternative software and hardware solutions that exploit physical effects for computation. According to machine learning theory, a neural…
Intelligent systems must deploy internal representations that are simultaneously structured -- to support broad generalization -- and selective -- to preserve input identity. We expose a fundamental limit on this tradeoff. For any model…
Computable Information Density (CID), the ratio of the length of a losslessly compressed data file to that of the uncompressed file, is a measure of order and correlation in both equilibrium and nonequilibrium systems. Here we show that…
We present new results to model and understand the role of encoder-decoder design in machine learning (ML) from an information-theoretic angle. We use two main information concepts, information sufficiency (IS) and mutual information loss…
In problem-solving, we humans can come up with multiple novel solutions to the same problem. However, reinforcement learning algorithms can only produce a set of monotonous policies that maximize the cumulative reward but lack diversity and…
The realization of complex classification tasks requires training of deep learning (DL) architectures consisting of tens or even hundreds of convolutional and fully connected hidden layers, which is far from the reality of the human brain.…
A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has…
The Information Bottleneck (IB) principle facilitates effective representation learning by preserving label-relevant information while compressing irrelevant information. However, its strong reliance on accurate labels makes it inherently…
We introduce a novel, biologically plausible local learning rule that provably increases the robustness of neural dynamics to noise in nonlinear recurrent neural networks with homogeneous nonlinearities. Our learning rule achieves higher…
This paper proposes an information-theoretic representation learning framework, named conditional information flow maximization, to extract noise-invariant sufficient representations for the input data and target task. It promotes the…
Measurements acquired from distributed physical systems are often sparse and noisy. Therefore, signal processing and system identification tools are required to mitigate noise effects and reconstruct unobserved dynamics from limited sensor…
We develop a deterministic large-time mechanism yielding Ces{\`a}ro asymptotic observability inequalities from moving localized observations for conservative evolutions. On each observation interval, exact convexification on a compact…
Storage-efficient privacy-preserving learning is crucial due to increasing amounts of sensitive user data required for modern learning tasks. We propose a framework for reducing the storage cost of user data while at the same time providing…
We employ a unified framework for computing the information capacity of biological signaling systems using Fisher Information. By deriving closed-form or easily computable information capacity formulas, we quantify how well different…