An Optimal Time Variable Learning Framework for Deep Neural Networks
Optimization and Control
2022-04-20 v1 Machine Learning
Numerical Analysis
Numerical Analysis
Abstract
Feature propagation in Deep Neural Networks (DNNs) can be associated to nonlinear discrete dynamical systems. The novelty, in this paper, lies in letting the discretization parameter (time step-size) vary from layer to layer, which needs to be learned, in an optimization framework. The proposed framework can be applied to any of the existing networks such as ResNet, DenseNet or Fractional-DNN. This framework is shown to help overcome the vanishing and exploding gradient issues. Stability of some of the existing continuous DNNs such as Fractional-DNN is also studied. The proposed approach is applied to an ill-posed 3D-Maxwell's equation.
Cite
@article{arxiv.2204.08528,
title = {An Optimal Time Variable Learning Framework for Deep Neural Networks},
author = {Harbir Antil and Hugo Díaz and Evelyn Herberg},
journal= {arXiv preprint arXiv:2204.08528},
year = {2022}
}