The Entropy of Floating-Point Numbers
Information Theory
2026-05-13 v1 math.IT
Abstract
Here we present an analytic approximation for the entropy of floating-point numbers, along with bounds on the error of this approximation. It is well-known that the differential entropy is tightly linked to the discrete entropy of a uniformly quantized random variable. Our approximation uncovers a different quantity that provides this link for floating-point quantization. Additionally, we prove that the entropy of a floating-point quantized random variable is approximately unchanged under scaling. Closed-form expressions for the floating-point entropy of common distributions are provided and compared to exact results.
Keywords
Cite
@article{arxiv.2605.11546,
title = {The Entropy of Floating-Point Numbers},
author = {Sultan Daniels and Samuel H. D'Ambrosia and Michael R. DeWeese and Anant Sahai},
journal= {arXiv preprint arXiv:2605.11546},
year = {2026}
}
Comments
17 pages, 12 figures