Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information
Machine Learning
2026-05-13 v2 Machine Learning
Abstract
Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.
Keywords
Cite
@article{arxiv.2505.17907,
title = {Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information},
author = {Ka Long Keith Ho and Yoshinari Takeishi and Junichi Takeuchi},
journal= {arXiv preprint arXiv:2505.17907},
year = {2026}
}
Comments
18 pages, 1 figure, 1 table