The kernel-balanced equation for deep neural networks
Abstract
Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results in instability in our case.
Cite
@article{arxiv.2309.07367,
title = {The kernel-balanced equation for deep neural networks},
author = {Kenichi Nakazato},
journal= {arXiv preprint arXiv:2309.07367},
year = {2023}
}