Skip to main content
SpringerLink
Log in

Application of deep learning to the diagnosis of cervical lymph node metastasis from thyroid cancer with CT: external validation and clinical utility for resident training

  • Head and Neck
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Purpose

This study aimed to validate a deep learning model’s diagnostic performance in using computed tomography (CT) to diagnose cervical lymph node metastasis (LNM) from thyroid cancer in a large clinical cohort and to evaluate the model’s clinical utility for resident training.

Methods

The performance of eight deep learning models was validated using 3838 axial CT images from 698 consecutive patients with thyroid cancer who underwent preoperative CT imaging between January and August 2018 (3606 and 232 images from benign and malignant lymph nodes, respectively). Six trainees viewed the same patient images (n = 242), and their diagnostic performance and confidence level (5-point scale) were assessed before and after computer-aided diagnosis (CAD) was included.

Results

The overall area under the receiver operating characteristics (AUROC) of the eight deep learning algorithms was 0.846 (range 0.784–0.884). The best performing model was Xception, with an AUROC of 0.884. The diagnostic accuracy, sensitivity, specificity, positive predictive value, and negative predictive value of Xception were 82.8%, 80.2%, 83.0%, 83.0%, and 80.2%, respectively. After introducing the CAD system, underperforming trainees received more help from artificial intelligence than the higher performing trainees (p = 0.046), and overall confidence levels significantly increased from 3.90 to 4.30 (p < 0.001).

Conclusion

The deep learning–based CAD system used in this study for CT diagnosis of cervical LNM from thyroid cancer was clinically validated with an AUROC of 0.884. This approach may serve as a training tool to help resident physicians to gain confidence in diagnosis.

Key Points

• A deep learning-based CAD system for CT diagnosis of cervical LNM from thyroid cancer was validated using data from a clinical cohort. The AUROC for the eight tested algorithms ranged from 0.784 to 0.884.

• Of the eight models, the Xception algorithm was the best performing model for the external validation dataset with 0.884 AUROC. The accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were 82.8%, 80.2%, 83.0%, 83.0%, and 80.2%, respectively.

• The CAD system exhibited potential to improve diagnostic specificity and accuracy in underperforming trainees (3 of 6 trainees, 50.0%). This approach may have clinical utility as a training tool to help trainees to gain confidence in diagnoses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

AI:

Artificial intelligence

AUROC:

Area under the receiver operating characteristic curve

CAD:

Computer-aided diagnosis

CNN:

Convolutional neural network

CT:

Computed tomography

FNA:

Fine-needle aspiration

LNM:

Lymph node metastasis

NPV:

Negative predictive value

PPV:

Positive predictive value

PTC:

Papillary thyroid carcinoma

US:

Ultrasonography

References

  1. Savadjiev P, Chong J, Dohan A et al (2019) Demystification of AI-driven medical image interpretation: past, present and future. Eur Radiol 29(3):1616–1624

    Article  Google Scholar 

  2. McBee MP, Awan OA, Colucci AT et al (2018) Deep learning in radiology. Acad Radiol 25:1472–1480

    Article  Google Scholar 

  3. Soffer S, Ben-Cohen A, Shimon O, Amitai MM, Greenspan H, Klang E (2019) Convolutional neural networks for radiologic images: a radiologist’s guide. Radiology 290:590–606

    Article  Google Scholar 

  4. Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800–809

    Article  Google Scholar 

  5. Yoo YJ, Ha EJ, Cho YJ, Kim HL, Han M, Kang SY (2018) A computer-aided diagnosis system for thyroid nodules on ultrasonography: initial clinical experience. Korean J Radiol 19(4):665–672

    Article  Google Scholar 

  6. Jeong EY, Kim HL, Ha EJ, Park SY, Cho YJ, Han M (2018) Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol 29(4):1978–1985

    Article  Google Scholar 

  7. Kim HL, Ha EJ, Han M (2019) Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol 45(10):2672–2678

    Article  Google Scholar 

  8. Ha EJ, Baek JH, Na DG (2019) Deep convolutional neural network models for the diagnosis of thyroid cancer. Lancet Oncol 2019;20(3):e130

    Article  Google Scholar 

  9. Shin JH, Baek JH, Chung J et al (2016) Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology Consensus Statement and Recommendations. Korean J Radiol 17:370–395

    Article  Google Scholar 

  10. Haugen BR (2017) 2015 American Thyroid Association Management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: what is new and what has changed? Cancer 123:372–381

    Article  Google Scholar 

  11. Suh CH, Baek JH, Choi YJ, Lee JH (2017) Performance of CT in the preoperative diagnosis of cervical lymph node metastasis in patients with papillary thyroid cancer: a systematic review and meta-analysis. AJNR Am J Neuroradiol 38:154–161

    Article  CAS  Google Scholar 

  12. Lee Y, Kim JH, Baek JH et al (2018) Value of CT added to ultrasonography for the diagnosis of lymph node metastasis in patients with thyroid cancer. Head Neck 40:2137–2148

    Article  Google Scholar 

  13. Lee JH, Ha EJ, Kim JH (2019) Application of deep learning to the diagnosis of cervical lymph node metastasis from thyroid cancer with CT. Eur Radiol 29:5452–5457

    Article  Google Scholar 

  14. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image. arXiv1409.1556

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778

  16. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1251–1258

  17. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2818–2826

  18. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence

  19. Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T, Keutzer K (2014) Densenet: Implementing efficient convnet descriptor pyramids. ArXiv Preprint arXiv1404.1869

  20. Russakovsky O, Deng J, Su H et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115:211–252

    Article  Google Scholar 

  21. Grinberg M (2018) Flask web development: developing web applications with python. O’Reilly Media

  22. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2921–2929

  23. Selvaraju RR, Cogswell M, Das A, Vedantam R (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp 618–626

  24. Giger ML (2018) Machine learning in medical imaging. J Am Coll Radiol 15:512–520

    Article  Google Scholar 

  25. Choi YJ, Baek JH, Park HS et al (2017) A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid 27:546–552

    Article  Google Scholar 

Download references

Funding

This study has received funding by the National Research Foundation of Korea (no. 2017R1C1B5016217).

Author information

Authors and Affiliations

  1. Division of Biomedical Informatics, Seoul National University Biomedical Informatics (SNUBI), Seoul National University College of Medicine, Seoul, 110799, South Korea

    Jeong Hoon Lee

  2. Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 443-380, South Korea

    Eun Ju Ha, DaYoung Kim, Yong Jun Jung, Subin Heo, Yong-ho Jang, Sung Hyun An & Kyungmin Lee

Authors
  1. Jeong Hoon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eun Ju Ha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. DaYoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong Jun Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Subin Heo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yong-ho Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sung Hyun An
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kyungmin Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eun Ju Ha.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Eun Ju Ha.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Methodology

Retrospective

Diagnostic or prognostic study

Performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, J.H., Ha, E.J., Kim, D. et al. Application of deep learning to the diagnosis of cervical lymph node metastasis from thyroid cancer with CT: external validation and clinical utility for resident training. Eur Radiol 30, 3066–3072 (2020). https://doi.org/10.1007/s00330-019-06652-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-019-06652-4

Keywords

Search

Navigation