Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
Abstract
Multimodal networks have demonstrated remarkable performance improvements over their unimodal counterparts. Existing multimodal networks are designed in a multi-branch fashion that, due to the reliance on fusion strategies, exhibit deteriorated performance if one or more modalities are missing. In this work, we propose a modality invariant multimodal learning method, which is less susceptible to the impact of missing modalities. It consists of a single-branch network sharing weights across multiple modalities to learn inter-modality representations to maximize performance as well as robustness to missing modalities. Extensive experiments are performed on four challenging datasets including textual-visual (UPMC Food-101, Hateful Memes, Ferramenta) and audio-visual modalities (VoxCeleb1). Our proposed method achieves superior performance when all modalities are present as well as in the case of missing modalities during training or testing compared to the existing state-of-the-art methods.
Cite
@article{arxiv.2408.07445,
title = {Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach},
author = {Muhammad Saad Saeed and Shah Nawaz and Muhammad Zaigham Zaheer and Muhammad Haris Khan and Karthik Nandakumar and Muhammad Haroon Yousaf and Hassan Sajjad and Tom De Schepper and Markus Schedl},
journal= {arXiv preprint arXiv:2408.07445},
year = {2024}
}