Testing predictions of representation cost theory with CNNs
Abstract
It is widely acknowledged that trained convolutional neural networks (CNNs) have different levels of sensitivity to signals of different frequency. In particular, a number of empirical studies have documented CNNs sensitivity to low-frequency signals. In this work we show with theory and experiments that this observed sensitivity is a consequence of the frequency distribution of natural images, which is known to have most of its power concentrated in low-to-mid frequencies. Our theoretical analysis relies on representations of the layers of a CNN in frequency space, an idea that has previously been used to accelerate computations and study implicit bias of network training algorithms, but to the best of our knowledge has not been applied in the domain of model robustness.
Keywords
Cite
@article{arxiv.2210.01257,
title = {Testing predictions of representation cost theory with CNNs},
author = {Charles Godfrey and Elise Bishoff and Myles Mckay and Davis Brown and Grayson Jorgenson and Henry Kvinge and Eleanor Byler},
journal= {arXiv preprint arXiv:2210.01257},
year = {2023}
}
Comments
Comments welcome! V2: Conjecture on non-commutative generalized H\"older upgraded to Lemma 4.11, as a consequence restrictions on Theorem 4.9 removed, more datasets, more variable frequency statistics and more CNN architectures. V3: title updated to better reflect content, some new ablations with untrained networks