Skip to main content
SpringerLink
Log in

Evaluation of a convolutional neural network for ovarian tumor differentiation based on magnetic resonance imaging

  • Imaging Informatics and Artificial Intelligence
  • Published:
European Radiology Aims and scope Submit manuscript

A Correction to this article was published on 26 April 2021

This article has been updated

Abstract

Objectives

There currently lacks a noninvasive and accurate method to distinguish benign and malignant ovarian lesion prior to treatment. This study developed a deep learning algorithm that distinguishes benign from malignant ovarian lesion by applying a convolutional neural network on routine MR imaging.

Methods

Five hundred forty-five lesions (379 benign and 166 malignant) from 451 patients from a single institution were divided into training, validation, and testing set in a 7:2:1 ratio. Model performance was compared with four junior and three senior radiologists on the test set.

Results

Compared with junior radiologists averaged, the final ensemble model combining MR imaging and clinical variables had a higher test accuracy (0.87 vs 0.64, p < 0.001) and specificity (0.92 vs 0.64, p < 0.001) with comparable sensitivity (0.75 vs 0.63, p = 0.407). Against the senior radiologists averaged, the final ensemble model also had a higher test accuracy (0.87 vs 0.74, p = 0.033) and specificity (0.92 vs 0.70, p < 0.001) with comparable sensitivity (0.75 vs 0.83, p = 0.557). Assisted by the model’s probabilities, the junior radiologists achieved a higher average test accuracy (0.77 vs 0.64, Δ = 0.13, p < 0.001) and specificity (0.81 vs 0.64, Δ = 0.17, p < 0.001) with unchanged sensitivity (0.69 vs 0.63, Δ = 0.06, p = 0.302). With the AI probabilities, the junior radiologists had higher specificity (0.81 vs 0.70, Δ = 0.11, p = 0.005) but similar accuracy (0.77 vs 0.74, Δ = 0.03, p = 0.409) and sensitivity (0.69 vs 0.83, Δ = -0.146, p = 0.097) when compared with the senior radiologists.

Conclusions

These results demonstrate that artificial intelligence based on deep learning can assist radiologists in assessing the nature of ovarian lesions and improve their performance.

Key Points

• Artificial Intelligence based on deep learning can assess the nature of ovarian lesions on routine MRI with higher accuracy and specificity than radiologists.

• Assisted by the deep learning model’s probabilities, junior radiologists achieved better performance that matched those of senior radiologists.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Change history

Abbreviations

AI:

Artificial intelligence

AUC:

Area under the curve

MRI:

Magnetic resonance imaging

PR:

Precision-recall curve

ROC:

Receiver operating characteristic curve

US:

Ultrasound

References

  1. Siegel RL, Miller KD, Jemal A (2017) Cancer statistics, 2017. CA Cancer J Clin 67(1):7–30

    Article  Google Scholar 

  2. Levine D, Brown DL, Andreotti RF et al (2010) Management of asymptomatic ovarian and other adnexal cysts imaged at US Society of radiologists in ultrasound consensus conference statement. Ultrasound Q 26(3):121–131

    Article  Google Scholar 

  3. Patel MD, Ascher SM, Paspulati RM et al (2013) Managing incidental findings on abdominal and pelvic CT and MRI, part 1: white paper of the ACR incidental findings committee II on adnexal findings. J Am Coll Radiol 10(9):675–681

    Article  Google Scholar 

  4. Jacobs IJ, Parmar M, Skates SJ, Menon U (2016) Ovarian cancer screening: UKCTOCS trial – authors’ reply. Lancet 387(10038):2603–2604

    Article  Google Scholar 

  5. van Nagell JR Jr., Hoff JT (2013) Transvaginal ultrasonography in ovarian cancer screening: current perspectives. Int J Womens Health 6:25–33

  6. Sohaib SA, Reznek RH (2007) MR imaging in ovarian cancer. Cancer Imaging 7 Spec No A:S119–S129

    Article  CAS  Google Scholar 

  7. Siegelman ES, Outwater EK (1999) Tissue characterization in the female pelvis by means of MR imaging. Radiology. 212(1):5–18

    Article  CAS  Google Scholar 

  8. Pi S, Cao R, Qiang JW, Guo YH (2018) Utility of DWI with quantitative ADC values in ovarian tumors: a meta-analysis of diagnostic test performance. Acta Radiol 59(11):1386–1394

    Article  Google Scholar 

  9. Kombächer P, Hamm B, Becker R, Hese S, Weitzel HK, Wolf KJ (1992) Tumors of the adnexa--a comparison of magnetic resonance tomography, endosonography and the histological findings. Rofo 156:303–308

  10. Park SY, Oh YT, Jung DC (2016) Differentiation between borderline and benign ovarian tumors: combined analysis of MRI with tumor markers for large cystic masses (>/=5 cm). Acta Radiol 57(5):633–639

    Article  Google Scholar 

  11. Moore BJ, Steiner CA, Davis PH, Stocks C, Barrett ML (2006) Trends in hysterectomies and oophorectomies in hospital inpatient and ambulatory settings, 2005–2013: Statistical Brief #214. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD)

  12. Lass A (1999) The fertility potential of women with a single ovary. Hum Reprod Update 5(5):546–550

    Article  CAS  Google Scholar 

  13. Parker WH, Broder MS, Liu Z, Shoupe D, Farquhar C, Berek JS (2005) Ovarian conservation at the time of hysterectomy for benign disease. Obstet Gynecol 106(2):219–226

  14. Bi WL, Hosny A, Schabath MB et al (2019) Artificial intelligence in cancer imaging: clinical challenges and applications. CA Cancer J Clin 69(2):127–157

    PubMed  PubMed Central  Google Scholar 

  15. Chang K, Beers AL, Bai HX et al (2019) Automatic assessment of glioma burden: a deep learning algorithm for fully automated volumetric and bidimensional measurement. Neuro Oncol 21(11):1412–1422

    Article  Google Scholar 

  16. Papp L, Potsch N, Grahovac M et al (2018) Glioma survival prediction with combined analysis of in vivo (11)C-MET PET features, ex vivo features, and patient features by supervised machine learning. J Nucl Med 59(6):892–899

    Article  CAS  Google Scholar 

  17. Zhang X, Yan LF, Hu YC et al (2017) Optimizing a machine learning based glioma grading system using multi-parametric MRI histogram and texture features. Oncotarget. 8(29):47816–47830

    Article  Google Scholar 

  18. Chang K, Bai HX, Zhou H et al (2018) Residual convolutional neural network for the determination of IDH status in low- and high-grade gliomas from MR imaging. Clin Cancer Res 24(5):1073–1081

    Article  CAS  Google Scholar 

  19. Meinhold-Heerlein I, Fotopoulou C, Harter P et al (2016) The new WHO classification of ovarian, fallopian tube, and primary peritoneal cancer and its clinical implications. Arch Gynecol Obstet 293(4):695–700

    Article  Google Scholar 

  20. Zafar HM, Chadalavada SC, Kahn CE Jr et al (2015) Code abdomen: an assessment coding scheme for abdominal imaging findings possibly representing cancer. J Am Coll Radiol 12(9):947–950

    Article  Google Scholar 

  21. Zwanenburg A, Leger S, Vallières M, Löck S (2016) Image biomarker standardisation initiative. arXiv preprint arXiv:161207003

  22. Agresti A, Coull BA (1998) Approximate is better than “exact” for interval estimation of binomial proportions. Am Stat 52(2):119–126

    Google Scholar 

  23. Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Stat 7(1):1–26

    Article  Google Scholar 

  24. Krippendorf K (1989) Content analysis, vol 1. Oxford University Press, New York

    Google Scholar 

  25. Vallieres M, Freeman CR, Skamene SR, El Naqa I (2015) A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Phys Med Biol 60(14):5471–5496

    Article  CAS  Google Scholar 

  26. Vallieres M, Kay-Rivest E, Perrin LJ et al (2017) Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci Rep 7(1):10117

    Article  Google Scholar 

  27. Olson RS, Urbanowicz RJ, Andrews PC, Lavender NA, Moore JH (2016) Automating biomedical data science through tree-based pipeline optimization. Paper presented at: European Conference on the Applications of Evolutionary Computation

  28. Medeiros LR, Freitas LB, Rosa DD et al (2011) Accuracy of magnetic resonance imaging in ovarian tumor: a systematic quantitative review. Am J Obstet Gynecol 204(1):67 e61–67 e10

    Article  Google Scholar 

  29. Medeiros LR, Rosa DD, da Rosa MI, Bozzetti MC (2009) Accuracy of ultrasonography with color Doppler in ovarian tumor: a systematic quantitative review. Int J Gynecol Cancer 19(7):1214–1220

    Article  Google Scholar 

  30. Riccio TJ, Adams HG, Munzing DE, Mattrey RF (1990) Magnetic resonance imaging as an adjunct to sonography in the evaluation of the female pelvis. Magn Reson Imaging 8(6):699–704

    Article  CAS  Google Scholar 

  31. Hricak H, Chen M, Coakley FV et al (2000) Complex adnexal masses: detection and characterization with MR imaging--multivariate analysis. Radiology. 214(1):39–46

    Article  CAS  Google Scholar 

  32. Martinez-Mas J, Bueno-Crespo A, Khazendar S et al (2019) Evaluation of machine learning methods with Fourier transform features for classifying ovarian tumors based on ultrasound images. PLoS One 14(7):e0219388

    Article  CAS  Google Scholar 

  33. Li Y, Jian J, Pickhardt PJ et al (2020) MRI-based machine learning for differentiating borderline from malignant epithelial ovarian tumors: a multicenter study. J Magn Reson Imaging 52:897–904. https://doi.org/10.1002/jmri.27084

  34. Deng J, Dong W, Socher R et al (2009) ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami 2–9.

  35. Xu Y, Hosny A, Zeleznik R et al (2019) Deep learning predicts lung cancer treatment response from serial medical imaging. Clin Cancer Res 25(11):3266–3275

    Article  Google Scholar 

  36. Hamm CA, Wang CJ, Savic LJ et al (2019) Deep learning for liver tumor diagnosis part I: development of a convolutional neural network classifier for multi-phasic MRI. Eur Radiol 29(7):3338–3347

    Article  Google Scholar 

  37. Zhang N, Yang G, Gao Z et al (2019) Deep learning for diagnosis of chronic myocardial infarction on nonenhanced cardiac cine MRI. Radiology. 291(3):606–617

    Article  Google Scholar 

  38. Soni P, Vashisht S (2019) Image segmentation for detecting polycystic ovarian disease using deep neural networks. Int J Comput Civ Struct Eng 7(3)

  39. Li W, Chu C, Cui Y, Zhang P, Zhu M (2012) Diffusion-weighted MRI: a useful technique to discriminate benign versus malignant ovarian surface epithelial tumors with solid and cystic components. Abdom Imaging 37(5):897–903

    Article  Google Scholar 

  40. Zhao SH, Qiang JW, Zhang GF et al (2014) Diffusion-weighted MR imaging for differentiating borderline from malignant epithelial tumours of the ovary: pathological correlation. Eur Radiol 24(9):2292–2299

    Article  Google Scholar 

  41. Abascal-Saiz A, Sotillo-Mallo L, de Santiago J, Zapardiel I (2014) Management of borderline ovarian tumours: a comprehensive review of the literature. Ecancermedicalscience. 8:403

    PubMed  PubMed Central  Google Scholar 

  42. Roth HR, Lu L, Seff A et al (2014) A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014. MICCAI 2014. Lecture Notes in Computer Science, vol 8673

  43. Tu X, Xie M, Gao J et al (2017) Automatic categorization and scoring of solid, part-solid and non-solid pulmonary nodules in CT images with convolutional neural network. Sci Rep 7(1):8533. https://doi.org/10.1038/s41598-017-08040-8

Download references

Acknowledgments

We would like to acknowledge and thank Huizhou Li (HZL) and Dan Li (DL) for performing the segmentation of MR images on the test set. We would like to thank Hui Liu (HL), Ting Huang (TH), Xin Su (XS), and Yijun Zhao (YJZ) for evaluating the test set as junior radiologists, and Dehong Peng (DHP), Shuanglin Zeng (SLZ), and Juan Chen (JC) for evaluating the test set as senior radiologists.

Funding

This work was supported by RSNA Research Scholar Grant to H. Bai, by a training grant from the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health (5T32EB1680) to K. Chang and by the National Cancer Institute (NCI) of the National Institutes of Health (F30CA239407) to K. Chang. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. TL was partially supported by Institutional Development Award Number U54GM115677 from the National Institute of General Medical Sciences of the National Institutes of Health which funds Advance Clinical and Translational Research (Advance-CTR).

Author information

Author notes

  1. Robin Wang and Yeyu Cai contributed equally to this work.

Authors and Affiliations

  1. Department of Radiology, the Second Xiangya Hospital, Central South University, Changsha, China

    Robin Wang, Yeyu Cai, Enhua Xiao & Jing Wu

  2. Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA

    Robin Wang, Iris K. Lee & Zishu Zhang

  3. Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA

    Iris K. Lee

  4. School of Computer Science and Engineering, Central South University, Changsha, China

    Rong Hu

  5. Department of Diagnostic Imaging, Rhode Island Hospital and Alpert Medical School of Brown University, Providence, RI, USA

    Subhanik Purkayastha & Harrison X. Bai

  6. Warren Alpert Medical School at Brown University, Providence, RI, USA

    Ian Pan, Thomas Yi & Thi My Linh Tran

  7. Department of Pathology, Rhode Island Hospital and Alpert Medical School of Brown University, Providence, RI, USA

    Shaolei Lu

  8. Department of Biostatistics, Center for Statistical Sciences, Brown University School of Public Health, Providence, RI, USA

    Tao Liu

  9. Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, MA, USA

    Ken Chang

  10. Department of Radiology, Brigham and Women’s Hospital, Boston, MA, USA

    Raymond Y. Huang

  11. Department of Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA

    Paul J. Zhang

Authors
  1. Robin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yeyu Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Iris K. Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Subhanik Purkayastha
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ian Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Thomas Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Thi My Linh Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Shaolei Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ken Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Raymond Y. Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Paul J. Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Zishu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Enhua Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Jing Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Harrison X. Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jing Wu or Harrison X. Bai.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Harrison Bai.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Methodology

• Retrospective

• case-control study

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: Robin Wang is affiliated with two institutions.

Electronic supplementary material

ESM 1

(DOCX 97 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, R., Cai, Y., Lee, I.K. et al. Evaluation of a convolutional neural network for ovarian tumor differentiation based on magnetic resonance imaging. Eur Radiol 31, 4960–4971 (2021). https://doi.org/10.1007/s00330-020-07266-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-020-07266-x

Keywords

Search

Navigation