受神经启发的层次化多模态学习
机器学习
2024-04-24 v3 人工智能
摘要
整合与处理来自多种来源或模态的信息，对于获得对现实世界全面而准确的感知至关重要。受神经科学启发，我们提出利用信息瓶颈概念的信息论层次化感知（ITHP）模型。与多数旨在将所有模态作为输入的传统融合模型不同，我们的模型将主要模态指定为输入，其余模态作为信息通路中的检测器。我们提出的感知模型聚焦于通过平衡潜状态与输入模态状态间互信息的最小化，以及潜状态与剩余模态状态间互信息的最大化，构建高效紧凑的信息流。该方法产生保留相关信息同时最小化冗余的紧凑潜状态表示，从而显著提升下游任务性能。在MUStARD与CMU-MOSI数据集上的实验评估表明，我们的模型在多模态学习场景中持续提炼关键信息，优于当前最优基准。
引用
@article{arxiv.2309.15877,
title = {Neuro-Inspired Hierarchical Multimodal Learning},
author = {Xiongye Xiao and Gengshuo Liu and Gaurav Gupta and Defu Cao and Shixuan Li and Yaxing Li and Tianqing Fang and Mingxi Cheng and Paul Bogdan},
journal= {arXiv preprint arXiv:2309.15877},
year = {2024}
}
备注
I am requesting the withdrawal of this submission due to an inadvertent duplication. The paper was submitted twice under different IDs, which was not intentional. The other submission (arXiv:2404.09403) contains the most updated and comprehensive version of the paper, and I would like to retain that as the sole version on the platform