Training and Generating Neural Networks in Compressed Weight Space
Machine Learning
2022-01-03 v1 Computation and Language
Abstract
The inputs and/or outputs of some neural nets are weight matrices of other neural nets. Indirect encodings or end-to-end compression of weight matrices could help to scale such approaches. Our goal is to open a discussion on this topic, starting with recurrent neural networks for character-level language modelling whose weight matrices are encoded by the discrete cosine transform. Our fast weight version thereof uses a recurrent neural network to parameterise the compressed weights. We present experimental results on the enwik8 dataset.
Cite
@article{arxiv.2112.15545,
title = {Training and Generating Neural Networks in Compressed Weight Space},
author = {Kazuki Irie and Jürgen Schmidhuber},
journal= {arXiv preprint arXiv:2112.15545},
year = {2022}
}
Comments
Presented at ICLR 2021 Workshop on Neural Compression, https://openreview.net/forum?id=qU1EUxdVd_D