Breast cancer is the most common invasive cancer in women. Besides the primary B-mode ultrasound screening, sonographers have explored the inclusion of Doppler, strain and shear-wave elasticity imaging to advance the diagnosis. However, recognizing useful patterns in all types of images and weighing up the significance of each modality can elude less-experienced clinicians. In this paper, we explore, for the first time, an automatic way to combine the four types of ultrasonography to discriminate between benign and malignant breast nodules. A novel multimodal network is proposed, along with promising learnability and simplicity to improve classification accuracy. The key is using a weight-sharing strategy to encourage interactions between modalities and adopting an additional cross-modalities objective to integrate global information. In contrast to hardcoding the weights of each modality in the model, we embed it in a Reinforcement Learning framework to learn this weighting in an end-to-end manner. Thus the model is trained to seek the optimal multimodal combination without handcrafted heuristics. The proposed framework is evaluated on a dataset contains 1616 set of multimodal images. Results showed that the model scored a high classification accuracy of 95.4%, which indicates the efficiency of the proposed method.
@article{arxiv.2008.03435,
title = {Auto-weighting for Breast Cancer Classification in Multimodal Ultrasound},
author = {Wang Jian and Miao Juzheng and Yang Xin and Li Rui and Zhou Guangquan and Huang Yuhao and Lin Zehui and Xue Wufeng and Jia Xiaohong and Zhou Jianqiao and Huang Ruobing and Ni Dong},
journal= {arXiv preprint arXiv:2008.03435},
year = {2020}
}