Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.
@article{arxiv.2205.11833,
title = {Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model},
author = {Sosuke Kobayashi and Shun Kiyono and Jun Suzuki and Kentaro Inui},
journal= {arXiv preprint arXiv:2205.11833},
year = {2022}
}
Comments
Workshop on Challenges & Perspectives in Creating Large Language Models (BigScience) 2022