Joint Detection and Diagnosis of Prostate Cancer in Multi-parametric MRI Based on Multimodal Convolutional Neural Networks

  • Xin YangEmail author
  • Zhiwei Wang
  • Chaoyue Liu
  • Hung Minh Le
  • Jingyu Chen
  • Kwang-Ting (Tim) Cheng
  • Liang WangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10435)


This paper presents an automated method for jointly localizing prostate cancer (PCa) in multi-parametric MRI (mp-MRI) images and assessing the aggressiveness of detected lesions. Our method employs multimodal multi-label convolutional neural networks (CNNs), which are trained in a weakly-supervised manner by providing a set of prostate images with image-level labels without priors of lesions’ locations. By distinguishing images with different labels, discriminative visual patterns related to indolent PCa and clinically significant (CS) PCa are automatically learned from clutters of prostate tissues. Cancer response maps (CRMs) with each pixel indicating the likelihood of being part of indolent/CS are explicitly generated at the last convolutional layer. We define new back-propagate error of CNN to enforce both optimized classification results and consistent CRMs for different modalities. Our method enables the feature learning processes of different modalities to mutually influence each other and, in turn yield more representative features. Comprehensive evaluation based on 402 lesions demonstrates superior performance of our method to the state-of-the-art method [13].


Prostate cancer detection and diagnosis Convolutional neural network Multimodal fusion 



This work is funded by National Natural Science Foundation of China: 61502188.


  1. 1.
    Artan, Y., Haider, M.A., Langer, D.L., van der Kwast, T.H., Evans, A.J., Yang, Y., Wernick, M.N., Trachtenberg, J., Yetik, I.S.: Prostate cancer localization with multispectral MRI using cost-sensitive support vector machines and conditional random fields. TIP 19(9), 2444–2455 (2010)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Fehr, D., Veeraraghavan, H., Wibmer, A., Gondo, T., Matsumoto, K., Vargas, H.A., Sala, E., Hricak, H., Deasy, J.O.: Automatic classification of prostate cancer gleason scores from multiparametric magnetic resonance images. Proc. NAS 112(46), E6265–E6273 (2015)CrossRefGoogle Scholar
  3. 3.
    Lemaitre, G.: Computer-Aided Diagnosis for Prostate Cancer using Multi-Parametric Magnetic Resonance Imaging. Ph.D. thesis, Universite de Bourgogne; Universitat de Girona (2016)Google Scholar
  4. 4.
    Litjens, G., Debats, O., Barentsz, J., Karssemeijer, N., Huisman, H.: Computer-aided detection of prostate cancer in MRI. TMI 33(5), 1083–1092 (2014)Google Scholar
  5. 5.
    Niaf, E., Flamary, R., Rouviere, O., Lartizien, C., Canu, S.: Kernel-based learning from both qualitative and quantitative labels: application to prostate cancer diagnosis based on multiparametric MR imaging. TIP 23(3), 979–991 (2014)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Otsu, N.: A threshold selection method from gray-level histograms. Automatica 11(285–296), 23–27 (1975)Google Scholar
  7. 7.
    Peng, Y., Jiang, Y., Yang, C., Brown, J.B., Antic, T., Sethi, I., Schmid-Tannwald, C., Giger, M.L., Eggener, S.E., Oto, A.: Quantitative analysis of multiparametric prostate MR images: differentiation between prostate cancer and normal tissue and correlation with gleason scorea computer-aided diagnosis development study. Radiology 267(3), 787–796 (2013)CrossRefGoogle Scholar
  8. 8.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi: 10.1007/978-3-319-24574-4_28 CrossRefGoogle Scholar
  9. 9.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceeding of CVPR, pp. 1–9 (2015)Google Scholar
  10. 10.
    Tiwari, P., Kurhanewicz, J., Madabhushi, A.: Multi-kernel graph embedding for detection, gleason grading of prostate cancer via MRI/MRS. Med. Image Anal. 17(2), 219–235 (2013)CrossRefGoogle Scholar
  11. 11.
    Wang, S., Burtt, K., Turkbey, B., Choyke, P., Summers, R.M.: Computer aided-diagnosis of prostate cancer on multiparametric MRI: a technical review of current research. In: BioMed Research International 2014 (2014)Google Scholar
  12. 12.
    Xinran, Z., HungLe, M., Holden, W., Michael, K., Steven, R., William, H., Xin, Y., Kyunghyun, S.: Fine-tuned deep convolutional neural network for automatic detection of clinically significant prostate cancer with multi-parametric MRI. In: Proceeding of ISMRM (2017, to appear)Google Scholar
  13. 13.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceeding of CVPR (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Xin Yang
    • 1
    Email author
  • Zhiwei Wang
    • 1
  • Chaoyue Liu
    • 1
  • Hung Minh Le
    • 1
  • Jingyu Chen
    • 1
  • Kwang-Ting (Tim) Cheng
    • 2
  • Liang Wang
    • 3
    Email author
  1. 1.School of EICHUSTWuhanChina
  2. 2.School of EngineeringHKUSTKowloonHong Kong
  3. 3.Department of Radiology, Tongji HospitalHUSTWuhanChina

Personalised recommendations