Neural Networks Use Distance Metrics
Abstract
We present empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations. We independently manipulate both distance and intensity properties of internal activations in trained models, finding that both architectures are highly sensitive to small distance-based perturbations while maintaining robust performance under large intensity-based perturbations. These findings challenge the prevailing intensity-based interpretation of neural network activations and offer new insights into their learning and decision-making processes.
Cite
@article{arxiv.2411.17932,
title = {Neural Networks Use Distance Metrics},
author = {Alan Oursland},
journal= {arXiv preprint arXiv:2411.17932},
year = {2024}
}
Comments
8 pages excluding references and appendix. 12 pages total. 3 figures. The code for the experiments in this paper is available at https://github.com/alanoursland/neural_networks_use_distance_metrics