Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR
@article{arxiv.2109.01949,
title = {Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment},
author = {Zhanghexuan Ji and Mohammad Abuzar Shaikh and Dana Moukheiber and Sargur Srihari and Yifan Peng and Mingchen Gao},
journal= {arXiv preprint arXiv:2109.01949},
year = {2021}
}
Comments
10 Pages, 1 Figure, 3 Tables, Accepted in 12th Machine Learning in Medical Imaging (MLMI 2021) workshop