Probabilistic Examination of Least Squares Error in Low-bitwidth Cholesky Decomposition
Information Theory
2024-04-09 v1 math.IT
Abstract
In this paper, we propose a new approach to justify a round-off error impact on the accuracy of the linear least squares (LS) solution using Cholesky decomposition. This decomposition is widely employed to inverse a matrix in the linear detector of the Multi-User multi-antenna receiver. The proposed stochastic bound is much closer to actual errors than other numerical bounds. It was tested with a half-precision format and validated in realistic scenarios. Experimental results demonstrate our approach predicts errors very close to those achieved by simulations. The proposed approach can be employed to analyze the resulting round-off error in many other applications.
Keywords
Cite
@article{arxiv.2404.05082,
title = {Probabilistic Examination of Least Squares Error in Low-bitwidth Cholesky Decomposition},
author = {Alexander Osinsky and Roman Bychkov and Mikhail Trefilov and Vladimir Lyashev and Andrey Ivanov},
journal= {arXiv preprint arXiv:2404.05082},
year = {2024}
}