We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise compared to equivalent monolingual ones. We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. Our analysis connects robustness to spectral properties of the learned representation and proves that multitasking leads to higher robustness for diverse task vectors. We open-source our code and models: https://github.com/giannisdaras/multilingual_robustness
@article{arxiv.2210.11618,
title = {Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve},
author = {Giannis Daras and Negin Raoof and Zoi Gkalitsiou and Alexandros G. Dimakis},
journal= {arXiv preprint arXiv:2210.11618},
year = {2022}
}