Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters. Code available here.
@article{arxiv.2310.12808,
title = {Model Merging by Uncertainty-Based Gradient Matching},
author = {Nico Daheim and Thomas Möllenhoff and Edoardo Maria Ponti and Iryna Gurevych and Mohammad Emtiyaz Khan},
journal= {arXiv preprint arXiv:2310.12808},
year = {2024}
}