Generalist models, which are capable of performing diverse multi-modal tasks in a task-agnostic way within a single model, have been explored recently. Being, hopefully, an alternative to approaching general-purpose AI, existing generalist models are still at an early stage, where modality and task coverage is limited. To empower multi-modal task-scaling and speed up this line of research, we release a generalist model learning system, OFASys, built on top of a declarative task interface named multi-modal instruction. At the core of OFASys is the idea of decoupling multi-modal task representations from the underlying model implementations. In OFASys, a task involving multiple modalities can be defined declaratively even with just a single line of code. The system automatically generates task plans from such instructions for training and inference. It also facilitates multi-task training for diverse multi-modal workloads. As a starting point, we provide presets of 7 different modalities and 23 highly-diverse example tasks in OFASys, with which we also develop a first-in-kind, single model, OFA+, that can handle text, image, speech, video, and motion data. The single OFA+ model achieves 95% performance in average with only 16% parameters of 15 task-finetuned models, showcasing the performance reliability of multi-modal task-scaling provided by OFASys. Available at https://github.com/OFA-Sys/OFASys
Cite
@article{arxiv.2212.04408,
title = {OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models},
author = {Jinze Bai and Rui Men and Hao Yang and Xuancheng Ren and Kai Dang and Yichang Zhang and Xiaohuan Zhou and Peng Wang and Sinan Tan and An Yang and Zeyu Cui and Yu Han and Shuai Bai and Wenbin Ge and Jianxin Ma and Junyang Lin and Jingren Zhou and Chang Zhou},
journal= {arXiv preprint arXiv:2212.04408},
year = {2022}
}