Related papers: Information-Distilling Quantizers
Knowledge distillation is an effective way for model compression in deep learning. Given a large model (i.e., teacher model), it aims to improve the performance of a compact model (i.e., student model) by transferring the information from…
Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text…
We investigate performance limits and design of communication in the presence of uniform output quantization with moderate to high resolution. Under independent and identically distributed (i.i.d.) complex Gaussian codebook and nearest…
The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce…
This paper considers estimation of a quantized constant in noise when using uniform and nonuniform quantizers. Estimators based on simple arithmetic averages, on sample statistical moments and on the maximum-likelihood procedure are…
Quantum steering has recently been formalized in the framework of a resource theory of steering, and several quantifiers have already been introduced. Here, we propose an information-theoretic quantifier for steering called intrinsic…
Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods require similar tokenizers between the teacher and the student,…
A multi-resolution quantizer is a sequence of quantizers where the output of a coarser quantizer can be deduced from the output of a finer quantizer. In this paper, we propose an asymptotically scale-invariant multi-resolution quantizer,…
It is shown that, if the loss of entanglement along a quantum channel is sufficiently small, then approximate quantum error correction is possible, thereby generalizing what happens for coherent information. Explicit bounds are obtained for…
Knowledge distillation is widely used to improve generalization in practice, yet its theoretical understanding remains elusive. In the standard distillation setting, a teacher model provides soft predictions to guide the training of a…
Binary quantum information can be fault tolerantly encoded in states defined in infinite dimensional Hilbert spaces. Such states define a computational basis, and permit a perfect equivalence between continuous and discrete universal…
Quantum systems carry information. Quantum theory supports at least two distinct kinds of information (classical and quantum), and a variety of different ways to encode and preserve information in physical systems. A system's ability to…
This paper studies an information-theoretic one-shot variable-length secret key agreement problem with public discussion. Let $X$ and $Y$ be jointly distributed random variables, each taking values in some measurable space. Alice and Bob…
We consider the problem of multiple description scalar quantizers and describing the achievable rate-distortion tuples in that setting. We formulate it as a combinatorial optimization problem of arranging numbers in a matrix to minimize the…
The number of random bits required to approximate a target distribution in terms of un-normalized informational divergence is considered. It is shown that for a variable-to-variable length encoder, this number is lower bounded by the…
Encryption schemes often derive their power from the properties of the underlying algebra on the symbols used. Inspired by group theoretic tools, we use the centralizer of a subgroup of operations to present a private-key quantum…
Quantifying entanglement of multiple subsystems is a challenging open problem in interacting quantum systems. Here, we focus on two subsystems of length $\ell$ separated by a distance $r=\alpha\ell$ and quantify their entanglement…
With rapid increase in online information consumption, especially via social media sites, there have been concerns on whether people are getting selective exposure to a biased subset of the information space, where a user is receiving more…
The sampling, quantization, and estimation of a bounded dynamic-range bandlimited signal affected by additive independent Gaussian noise is studied in this work. For bandlimited signals, the distortion due to additive independent Gaussian…
Quantum entanglement plays crucial roles in quantum information processing. Quantum entangled states have become the key ingredient in the rapidly expanding field of quantum information science. Although the nonclassical nature of…