Rapid developments of AI tools are expected to offer unprecedented assistance to the research of natural science including chemistry. However, neither existing unimodal task-specific specialist models nor emerging general large multimodal models (LMM) can cover the wide range of chemical data modality and task categories. To address the real demands of chemists, a cross-modal Chemical General Intelligence (CGI) system, which serves as a truly practical and useful research assistant utilizing the great potential of LMMs, is in great need. In this work, we introduce the first Cross-modal Dialogue Foundation Model for Chemistry (ChemDFM-X). Diverse multimodal data are generated from an initial modality by approximate calculations and task-specific model predictions. This strategy creates sufficient chemical training corpora, while significantly reducing excessive expense, resulting in an instruction-tuning dataset containing 7.6M data. After instruction finetuning, ChemDFM-X is evaluated on extensive experiments of different chemical tasks with various data modalities. The results demonstrate the capacity of ChemDFM-X for multimodal and inter-modal knowledge comprehension. ChemDFM-X marks a significant milestone toward aligning all modalities in chemistry, a step closer to CGI.
@article{arxiv.2409.13194,
title = {ChemDFM-X: Towards Large Multimodal Model for Chemistry},
author = {Zihan Zhao and Bo Chen and Jingpiao Li and Lu Chen and Liyang Wen and Pengyu Wang and Zichen Zhu and Danyang Zhang and Ziping Wan and Yansi Li and Zhongyang Dai and Xin Chen and Kai Yu},
journal= {arXiv preprint arXiv:2409.13194},
year = {2025}
}