Convergence of backpropagation with momentum for network architectures with skip connections
Computer Vision and Pattern Recognition
2020-01-22 v4 Machine Learning
Abstract
We study a class of deep neural networks with networks that form a directed acyclic graph (DAG). For backpropagation defined by gradient descent with adaptive momentum, we show weights converge for a large class of nonlinear activation functions. The proof generalizes the results of Wu et al. (2008) who showed convergence for a feed forward network with one hidden layer. For an example of the effectiveness of DAG architectures, we describe an example of compression through an autoencoder, and compare against sequential feed forward networks under several metrics.
Keywords
Cite
@article{arxiv.1705.07404,
title = {Convergence of backpropagation with momentum for network architectures with skip connections},
author = {Chirag Agarwal and Joe Klobusicky and Dan Schonfeld},
journal= {arXiv preprint arXiv:1705.07404},
year = {2020}
}