Baichuan-Omni-1.5 Technical Report
Abstract
We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.
Keywords
Cite
@article{arxiv.2501.15368,
title = {Baichuan-Omni-1.5 Technical Report},
author = {Yadong Li and Jun Liu and Tao Zhang and Tao Zhang and Song Chen and Tianpeng Li and Zehuan Li and Lijun Liu and Lingfeng Ming and Guosheng Dong and Da Pan and Chong Li and Yuanbo Fang and Dongdong Kuang and Mingrui Wang and Chenglin Zhu and Youwei Zhang and Hongyu Guo and Fengyu Zhang and Yuran Wang and Bowen Ding and Wei Song and Xu Li and Yuqi Huo and Zheng Liang and Shusen Zhang and Xin Wu and Shuai Zhao and Linchu Xiong and Yozhen Wu and Jiahui Ye and Wenhao Lu and Bowen Li and Yan Zhang and Yaqi Zhou and Xin Chen and Lei Su and Hongda Zhang and Fuzhong Chen and Xuezhen Dong and Na Nie and Zhiying Wu and Bin Xiao and Ting Li and Shunya Dang and Ping Zhang and Yijia Sun and Jincheng Wu and Jinjie Yang and Xionghai Lin and Zhi Ma and Kegeng Wu and Jia li and Aiyuan Yang and Hui Liu and Jianqiang Zhang and Xiaoxi Chen and Guangwei Ai and Wentao Zhang and Yicong Chen and Xiaoqin Huang and Kun Li and Wenjing Luo and Yifei Duan and Lingling Zhu and Ran Xiao and Zhe Su and Jiani Pu and Dian Wang and Xu Jia and Tianyu Zhang and Mengyu Ai and Mang Wang and Yujing Qiao and Lei Zhang and Yanjun Shen and Fan Yang and Miao Zhen and Yijie Zhou and Mingyang Chen and Fei Li and Chenzheng Zhu and Keer Lu and Yaqi Zhao and Hao Liang and Youquan Li and Yanzhao Qin and Linzhuang Sun and Jianhua Xu and Haoze Sun and Mingan Lin and Zenan Zhou and Weipeng Chen},
journal= {arXiv preprint arXiv:2501.15368},
year = {2025}
}