Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
Abstract
Self-Supervised Learning (SSL) has been shown to learn useful and information-preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn hyper-representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.
Cite
@article{arxiv.2110.15288,
title = {Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction},
author = {Konstantin Schürholt and Dimche Kostadinov and Damian Borth},
journal= {arXiv preprint arXiv:2110.15288},
year = {2022}
}
Comments
Published at 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia. 31 Pages, 14 figures