Related papers: Entropy-Constrained Maximizing Mutual Information …
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a…
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to…
In this paper we find, for a class of bipartite quantum states, a nontrivial lower bound on the entropy gain resulting from the action of a tensor product of identity channel with an arbitrary channel. By means of that we then estimate…
Quantum capacity quantifies the amount of quantum information that can be transmitted by a quantum channel with an arbitrary small probability of error. Mathematically, the quantum capacity is given by an asymptotic formula involving the…
Current advancements in communication equipment demand the investigation of classical information transfer over quantum channels, by encompassing realistic scenarios in finite dimensions. To address this issue, we develop a framework for…
The entanglement-assisted classical capacity of a noisy quantum channel is the amount of information per channel use that can be sent over the channel in the limit of many uses of the channel, assuming that the sender and receiver have…
Motivated by recent high bandwidth communication systems, Inter-Symbol Interference (ISI) channels with 1-bit quantized output are considered under an average-power-constrained continuous input. While the exact capacity is difficult to…
Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we…
While the channel capacity reflects a theoretical upper bound on the achievable information transmission rate in the limit of infinitely many bits, it does not characterise the information transfer of a given encoding routine with finitely…
One of the most fundamental questions one can ask about a pair of random variables X and Y is the value of their mutual information. Unfortunately, this task is often stymied by the extremely large dimension of the variables. We might hope…
Maximum entropy estimation is of broad interest for inferring properties of systems across many different disciplines. In this work, we significantly extend a technique we previously introduced for estimating the maximum entropy of a set of…
The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission…
Neural-based image and video codecs are significantly more power-efficient when weights and activations are quantized to low-precision integers. While there are general-purpose techniques for reducing quantization effects, large losses can…
We study the problem of computing the capacity of a discrete memoryless channel under uncertainty affecting the channel law matrix, and possibly with a constraint on the average cost of the input distribution. The problem has been…
Motivated by communication systems with constrained complexity, we consider the problem of input symbol selection for discrete memoryless channels (DMCs). Given a DMC, the goal is to find a subset of its input alphabet, so that the optimal…
In this paper we address the issue of universal or robust communication over quantum channels. Specifically, we consider memoryless communication scenario with channel uncertainty which is an analog of compound channel in classical…
In this paper, a novel pattern classification approach is proposed by regularizing the classifier learning to maximize mutual information between the classification response and the true class label. We argue that, with the learned…
Quantum channels represent a broad spectrum of operations crucial to quantum information theory, encompassing everything from the transmission of quantum information to the manipulation of various resources. In the domain of states, the…
The work is devoted to study of quantum mutual information and coherent information -- the two important characteristics of quantum communication channel. Appropriate definitions of these quantities in the infinite-dimensional case are…
The optimal rate of reliable communication over a quantum channel can be enhanced by pre-shared entanglement. Whereas the enhancement may be unbounded in infinite-dimensional settings even when the input power is constrained, a…