Information Theoretic Perspective on Representation Learning
Information Theory
2026-05-27 v2 Machine Learning
math.IT
Abstract
An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.
Cite
@article{arxiv.2601.11334,
title = {Information Theoretic Perspective on Representation Learning},
author = {Deborah Pereg and Michael Wand},
journal= {arXiv preprint arXiv:2601.11334},
year = {2026}
}