The Lottery Ticket Hypothesis (LTH) showed that by iteratively training a model, removing connections with the lowest global weight magnitude and rewinding the remaining connections, sparse networks can be extracted. This global comparison removes context information between connections within a layer. Here we study means for recovering some of this layer distributional context and generalise the LTH to consider weight importance values rather than global weight magnitudes. We find that given a repeatable training procedure, applying different importance metrics leads to distinct performant lottery tickets with little overlapping connections. This strongly suggests that lottery tickets are not unique
Cite
@article{arxiv.2302.11244,
title = {Considering Layerwise Importance in the Lottery Ticket Hypothesis},
author = {Benjamin Vandersmissen and Jose Oramas},
journal= {arXiv preprint arXiv:2302.11244},
year = {2023}
}