English

Mario: Multimodal Graph Reasoning with Large Language Models

Computer Vision and Pattern Recognition 2026-03-27 v2

Abstract

Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.

Keywords

Cite

@article{arxiv.2603.05181,
  title  = {Mario: Multimodal Graph Reasoning with Large Language Models},
  author = {Yuanfu Sun and Kang Li and Pengkang Guo and Jiajin Liu and Qiaoyu Tan},
  journal= {arXiv preprint arXiv:2603.05181},
  year   = {2026}
}

Comments

CVPR 2026

R2 v1 2026-07-01T11:04:55.280Z