Gradient representations in ReLU networks as similarity functions
Machine Learning
2021-10-27 v1 Artificial Intelligence
Abstract
Feed-forward networks can be interpreted as mappings with linear decision surfaces at the level of the last layer. We investigate how the tangent space of the network can be exploited to refine the decision in case of ReLU (Rectified Linear Unit) activations. We show that a simple Riemannian metric parametrized on the parameters of the network forms a similarity function at least as good as the original network and we suggest a sparse metric to increase the similarity gap.
Keywords
Cite
@article{arxiv.2110.13581,
title = {Gradient representations in ReLU networks as similarity functions},
author = {Dániel Rácz and Bálint Daróczy},
journal= {arXiv preprint arXiv:2110.13581},
year = {2021}
}
Comments
Accepted at 29th ESANN 2021, 6-8 October 2021, Belgium, 7 pages, 1 figure