Prostate cancer biopsy benefits from accurate fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images. In the past few years, convolutional neural networks (CNNs) have been proved powerful in extracting image features crucial for image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are quite limited in its ability to understand spatial correspondence between features, a task in which the self-attention mechanism excels. This paper aims to develop a self-attention mechanism specifically for cross-modal image registration. Our proposed cross-modal attention block effectively maps each of the features in one volume to all features in the corresponding volume. Our experimental results demonstrate that a CNN network designed with the cross-modal attention block embedded outperforms an advanced CNN network 10 times of its size. We also incorporated visualization techniques to improve the interpretability of our network. The source code of our work is available at https://github.com/DIAL-RPI/Attention-Reg .
@article{arxiv.2107.04548,
title = {Cross-modal Attention for MRI and Ultrasound Volume Registration},
author = {Xinrui Song and Hengtao Guo and Xuanang Xu and Hanqing Chao and Sheng Xu and Baris Turkbey and Bradford J. Wood and Ge Wang and Pingkun Yan},
journal= {arXiv preprint arXiv:2107.04548},
year = {2021}
}