English
Related papers

Related papers: Benchmarking Retrieval-Augmented Multimodal Genera…

200 papers

Document Question Answering (DocQA) is a very common task. Existing methods using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and Retrieval Augmented Generation (RAG) often prioritize information from a single…

Machine Learning · Computer Science 2025-03-19 Siwei Han , Peng Xia , Ruiyi Zhang , Tong Sun , Yun Li , Hongtu Zhu , Huaxiu Yao

Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. A key challenge with evaluating MMRAG is the paucity of high-quality datasets matching the question styles and…

Computation and Language · Computer Science 2024-10-07 Ian Wu , Sravan Jayanthi , Vijay Viswanathan , Simon Rosenberg , Sina Pakazad , Tongshuang Wu , Graham Neubig

Multimodal Retrieval-Augmented Generation (MRAG) enables Multimodal Large Language Models (MLLMs) to generate responses with external multimodal evidence, and numerous video-based MRAG benchmarks have been proposed to evaluate model…

Computation and Language · Computer Science 2025-10-13 Kaiwen Wei , Xiao Liu , Jie Zhang , Zijian Wang , Ruida Liu , Yuming Yang , Xin Xiao , Xiao Sun , Haoyang Zeng , Changzai Pan , Yidan Zhang , Jiang Zhong , Peijin Wang , Yingchao Feng

Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for…

Computer Vision and Pattern Recognition · Computer Science 2025-03-04 Jian Chen , Ruiyi Zhang , Yufan Zhou , Tong Yu , Franck Dernoncourt , Jiuxiang Gu , Ryan A. Rossi , Changyou Chen , Tong Sun

Large Vision-Language Models (LVLMs) have made remarkable strides in multimodal tasks such as visual question answering, visual grounding, and complex reasoning. However, they remain limited by static training data, susceptibility to…

Artificial Intelligence · Computer Science 2025-08-27 Chan-Wei Hu , Yueqi Wang , Shuo Xing , Chia-Ju Chen , Suofei Feng , Ryan Rossi , Zhengzhong Tu

Multimodal Retrieval-Augmented Generation (MRAG) enhances large language models (LLMs) by integrating multimodal data (text, images, videos) into retrieval and generation processes, overcoming the limitations of text-only…

Information Retrieval · Computer Science 2025-04-15 Lang Mei , Siyu Mo , Zhihan Yang , Chong Chen

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed…

Computation and Language · Computer Science 2025-05-27 Yangning Li , Yinghui Li , Xinyu Wang , Yong Jiang , Zhen Zhang , Xinran Zheng , Hui Wang , Hai-Tao Zheng , Philip S. Yu , Fei Huang , Jingren Zhou

Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) in knowledge-intensive tasks such as those from medical domain. However, the sensitive nature of the medical…

Computation and Language · Computer Science 2024-11-15 Nghia Trung Ngo , Chien Van Nguyen , Franck Dernoncourt , Thien Huu Nguyen

The evolution of digital manufacturing requires intelligent Question Answering (QA) systems that can seamlessly integrate and analyze complex multi-modal data, such as text, images, formulas, and tables. Conventional Retrieval Augmented…

Computational Engineering, Finance, and Science · Computer Science 2026-01-27 Yunqing Li , Zihan Dong , Farhad Ameri , Jianbang Zhang

Retrieval-augmented generation (RAG) systems have predominantly focused on text-based retrieval, limiting their effectiveness in handling visually-rich documents that encompass text, images, tables, and charts. To bridge this gap, we…

Information Retrieval · Computer Science 2025-05-07 Mingjun Xu , Zehui Wang , Hengxing Cai , Renxin Zhong

Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm for expanding the knowledge capacity of Multimodal Large Language Models (MLLMs) by incorporating external knowledge sources into the generation process, and has been…

Computer Vision and Pattern Recognition · Computer Science 2026-04-28 Xu Yuan , Liangbo Ning , Qingqing Ye , Wenqi Fan , Qing Li

Medical Visual Question Answering (MedVQA) enables natural language queries over medical images to support clinical decision-making and patient care. The MEDIQA-WV 2025 shared task addressed wound-care VQA, requiring systems to generate…

Computation and Language · Computer Science 2025-10-17 A H M Rezaul Karim , Ozlem Uzuner

Retrieval-augmented generation (RAG) has emerged as a promising paradigm for enhancing large language models (LLMs) on multi-hop question answering (QA), which requires reasoning over evidence from multiple documents. Current multi-hop RAG…

Computation and Language · Computer Science 2026-05-28 Yikai Zhu , Kunfeng Chen , Qihuang Zhong , Juhua Liu , Bo Du

Recent advancements in large language models (LLMs) and multi-modal LLMs have been remarkable. However, these models still rely solely on their parametric knowledge, which limits their ability to generate up-to-date information and…

Artificial Intelligence · Computer Science 2025-04-22 Zihan Ling , Zhiyao Guo , Yixuan Huang , Yi An , Shuai Xiao , Jinsong Lan , Xiaoyong Zhu , Bo Zheng

Recent advancements in retrieval-augmented generation (RAG) have significantly enhanced the ability of large language models (LLMs) to perform complex question-answering (QA) tasks. In this paper, we introduce MedBioRAG, a…

Computation and Language · Computer Science 2025-12-15 Seonok Kim

Multimodal document retrieval aims to identify and retrieve various forms of multimodal content, such as figures, tables, charts, and layout information from extensive documents. Despite its increasing popularity, there is a notable lack of…

Information Retrieval · Computer Science 2025-11-10 Kuicai Dong , Yujing Chang , Xin Deik Goh , Dexun Li , Ruiming Tang , Yong Liu

Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce…

Computer Vision and Pattern Recognition · Computer Science 2026-01-21 Peizhou Huang , Zixuan Zhong , Zhongwei Wan , Donghao Zhou , Samiul Alam , Xin Wang , Zexin Li , Zhihao Dou , Li Zhu , Jing Xiong , Chaofan Tao , Yan Xu , Dimitrios Dimitriadis , Tuo Zhang , Mi Zhang

Retrieval-Augmented Generation (RAG) has demonstrated remarkable success in enhancing Large Language Models (LLMs) through external knowledge integration, yet its application has primarily focused on textual content, leaving the rich domain…

Information Retrieval · Computer Science 2025-02-04 Xubin Ren , Lingrui Xu , Long Xia , Shuaiqiang Wang , Dawei Yin , Chao Huang

Multimodal Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, existing multimodal RAG systems predominantly rely on coarse-grained…

Information Retrieval · Computer Science 2026-05-25 Yifan Zhu , Yu Mi , Yue Lu , Yanchu Guan , Zhixuan Chu

Scientific research demands sophisticated reasoning over multimodal data, a challenge especially prevalent in biology. Despite recent advances in multimodal large language models (MLLMs) for AI-assisted research, existing multimodal…