Optimization without Backpropagation
Machine Learning
2022-09-15 v1 Optimization and Control
Abstract
Forward gradients have been recently introduced to bypass backpropagation in autodifferentiation, while retaining unbiased estimators of true gradients. We derive an optimality condition to obtain best approximating forward gradients, which leads us to mathematical insights that suggest optimization in high dimension is challenging with forward gradients. Our extensive experiments on test functions support this claim.
Keywords
Cite
@article{arxiv.2209.06302,
title = {Optimization without Backpropagation},
author = {Gabriel Belouze},
journal= {arXiv preprint arXiv:2209.06302},
year = {2022}
}
Comments
11 pages, 6 figures, associated implementation available at https://github.com/gbelouze/forward-gradient