Related papers: Information-Distilling Quantizers
The quadratic decaying property of the information rate function states that given a fixed conditional distribution $p_{\mathsf{Y}|\mathsf{X}}$, the mutual information between the (finite) discrete random variables $\mathsf{X}$ and…
Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem.…
Information leakage to a guessing adversary in index coding is studied, where some messages in the system are sensitive and others are not. The non-sensitive messages can be used by the server like secret keys to mitigate leakage of the…
Genuineness and distillability of entanglement play a key role in quantum information tasks, and they are easily disturbed by the noise. We construct a family of multipartite states without genuine entanglement and distillability sudden…
The distribution and processing of quantum entanglement form the basis of quantum communication and quantum computing. The realization of the two is difficult because quantum information inherently has a high susceptibility to decoherence,…
Recent work on distilling Whisper's knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. This results in small, efficient, and dedicated models. However, a critical step of…
We introduce the Mutual Information Machine (MIM), a probabilistic auto-encoder for learning joint distributions over observations and latent variables. MIM reflects three design principles: 1) low divergence, to encourage the encoder and…
In this paper a numerical method is presented, which finds a lower bound for the mutual information between a binary and an arbitrary finite random variable with joint distributions that have a variational distance not greater than a known…
In this paper, we examine the role of stochastic quantizers for privacy preservation. We first employ a static stochastic quantizer and investigate its corresponding privacy-preserving properties. Specifically, we demonstrate that a…
Vision generative models have recently made significant advancements along two primary paradigms: diffusion-style and language-style, both of which have demonstrated excellent scaling laws. Quantization is crucial for efficiently deploying…
Consider the problem where a statistician in a two-node system receives rate-limited information from a transmitter about marginal observations of a memoryless process generated from two possible distributions. Using its own observations,…
The notion of distillable entanglement is one of the fundamental concepts of quantum information theory. Unfortunately, there is an apparent mismatch between the intuitive and rigorous definitions of distillable entanglement. To be precise,…
Compared to traditional learning from scratch, knowledge distillation sometimes makes the DNN achieve superior performance. This paper provides a new perspective to explain the success of knowledge distillation, i.e., quantifying knowledge…
With the explosive growth of image databases, deep hashing, which learns compact binary descriptors for images, has become critical for fast image retrieval. Many existing deep hashing methods leverage quantization loss, defined as distance…
Around the turn of the century, Shor formulated his well-known and still-open conjecture stating that the accessible information of any quantum dichotomy, that is the maximum amount of classical information that can be decoded from a binary…
Artistic style transfer aims to transfer the style of an artwork to a photograph while maintaining its original overall content. Many prior works focus on designing various transfer modules to transfer the style statistics to the content…
Quantum Internet relies on quantum entanglement as a fundamental resource for secure and efficient quantum communication, reshaping data transmission. In this context, entanglement distillation emerges as a crucial process that plays a…
In this paper, we use quantization to construct a nonparametric estimator of conditional quantiles of a scalar response $Y$ given a d-dimensional vector of covariates $X$. First we focus on the population level and show how optimal…
Universal scaling of entanglement estimators of critical quantum systems has drawn a lot of attention in the past. Recent studies indicate that similar universal properties can be found for bipartite information estimators of classical…
Sharing entanglement across quantum interconnects is fundamental for quantum information processing. We discuss a practical setting where this interconnect, modeled by a quantum channel, is used once with the aim of sharing high fidelity…