Global Optimality of Elman-type RNN in the Mean-Field Regime
Machine Learning
2023-03-14 v1 Machine Learning
Abstract
We analyze Elman-type Recurrent Reural Networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime
Keywords
Cite
@article{arxiv.2303.06726,
title = {Global Optimality of Elman-type RNN in the Mean-Field Regime},
author = {Andrea Agazzi and Jianfeng Lu and Sayan Mukherjee},
journal= {arXiv preprint arXiv:2303.06726},
year = {2023}
}
Comments
31 pages, 2 figures