Approximation Capabilities of Neural ODEs and Invertible Residual Networks
Abstract
Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved. Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a -dimensional Euclidean space can be approximated by a Neural ODE operating on a -dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.
Keywords
Cite
@article{arxiv.1907.12998,
title = {Approximation Capabilities of Neural ODEs and Invertible Residual Networks},
author = {Han Zhang and Xi Gao and Jacob Unterman and Tom Arodz},
journal= {arXiv preprint arXiv:1907.12998},
year = {2020}
}