Skip to main content

An integrated AI model to improve diagnostic accuracy of ultrasound and output known risk features in suspicious thyroid nodules



From the viewpoint of ultrasound (US) physicians, an ideal thyroid US computer-assisted diagnostic (CAD) system for thyroid cancer should perform well in suspicious thyroid nodules with atypical risk features and be able to output explainable results. This study aims to develop an explainable US CAD model for suspicious thyroid nodules.


A total of 2992 solid or almost-solid thyroid nodules were analyzed retrospectively. All nodules had pathological results (1070 malignancies and 1992 benignities) confirmed by ultrasound-guided fine-needle aspiration cytology and histopathology after thyroidectomy. A deep learning model (ResNet50) and a multiple risk features learning ensemble model (XGBoost) were used to train the US images of 2794 thyroid nodules. Then, an integrated AI model was generated by combining both models. The diagnostic accuracies of the three AI models (ResNet50, XGBoost, and the integrated model) were predicted in a testing set including 198 thyroid nodules and compared to the diagnostic efficacy of five ultrasonographers.


The accuracy of the integrated model was 76.77%, while the mean accuracy of the ultrasonographers was 68.38%. Of the risk features, microcalcifications showed the highest contribution to the diagnosis of malignant nodules.


The integrated AI model in our study can improve the diagnostic accuracy of suspicious thyroid nodules and output the known risk features simultaneously, thus aiding in training young ultrasonographers by linking the explainable results to their clinical experience and advancing the acceptance of AI diagnosis for thyroid cancer in clinical practice.

Key Points

• We developed an artificial intelligence (AI) diagnosis model based on both deep learning and multiple risk feature ensemble learning methods.

• The AI diagnosis model showed higher diagnostic accuracy for suspicious thyroid nodules than ultrasonographers.

• The AI diagnosis model showed partial explainability by outputting the known risk features, thus aiding young ultrasonic doctors in increasing the diagnostic level for thyroid cancer.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5



The American College of Radiology


Artificial intelligence


The area under the curve


Deep learning


Fine-needle aspiration


Region of interest


Thyroid Image Radiology and Data System


US computer-assisted diagnosis




  1. Sung H, Ferlay J, Siegel RL et al (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 71(3):209–249

    Google Scholar 

  2. Li M, Dal Maso L, Vaccarella S (2020) Global trends in thyroid cancer incidence and the impact of overdiagnosis. Lancet Diabetes Endocrinol 8(6):468–470

    Article  Google Scholar 

  3. Durante C, Grani G, Lamartina L, Filetti S, Mandel SJ, Cooper DS (2018) The diagnosis and management of thyroid nodules: a review. JAMA 319(9):914–924

    Article  Google Scholar 

  4. Itani M, Assaker R, Moshiri M, Dubinsky TJ, Dighe MK (2019) Inter-observer variability in the American College of Radiology Thyroid Imaging Reporting and Data System: in-depth analysis and areas for improvement. Ultrasound Med Biol 45(2):461–470

    Article  Google Scholar 

  5. Vaccarella S, Franceschi S, Bray F, Wild CP, Plummer M, Dal Maso L (2016) Worldwide thyroid-cancer epidemic? The increasing impact of overdiagnosis. N Engl J Med 375(7):614–617

    Article  Google Scholar 

  6. Tessler FN, Middleton WD, Grant EG et al (2017) ACR Thyroid Imaging, Reporting and Data System (TI-RADS): white paper of the ACR TI-RADS Committee. J Am Coll Radiol 14(5):587–595

    Article  Google Scholar 

  7. Hoang JK, Middleton WD, Farjat AE et al (2018) Reduction in thyroid nodule biopsies and improved accuracy with American College of Radiology Thyroid Imaging Reporting and Data System. Radiology 287(1):185–193

    Article  Google Scholar 

  8. Buda M, Wildman-Tobriner B, Hoang JK et al (2019) Management of thyroid nodules seen on US images: deep learning may match performance of radiologists. Radiology 292(3):695–701

    Article  Google Scholar 

  9. Thomas J, Ledger GA, Mamillapalli CK (2020) Use of artificial intelligence and machine learning for estimating malignancy risk of thyroid nodules. Curr Opin Endocrinol Diabetes Obes 27(5):345–350

    Article  Google Scholar 

  10. Thomas J, Haertling T (2020) AIBx, Artificial intelligence model to risk stratify thyroid nodules. Thyroid 30(6):878–884

    Article  Google Scholar 

  11. Ye FY, Lyu GR, Li SQ et al (2021) Diagnostic performance of ultrasound computer-aided diagnosis software compared with that of radiologists with different levels of expertise for thyroid malignancy: a multicenter prospective study. Ultrasound Med Biol 47(1):114–124

    Article  Google Scholar 

  12. Morid MA, Borjali A, Del Fiol G (2021) A scoping review of transfer learning research on medical image analysis using ImageNet. Comput Biol Med 128:104115

    Article  Google Scholar 

  13. Liu SF, Wang Y, Yang X (2019) Deep learning in medical ultrasound analysis: a review. Engineering 5:261–275

    Article  Google Scholar 

  14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA. Available via Accessed 16 Aug 2021

  15. Krizhevsky A, Sutskever I, Hinton G (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  16. Shin JH, Baek JH, Chung J et al (2016) Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology Consensus Statement and Recommendations. Korean J Radiol 17(3):370–395

    PubMed Central  Article  Google Scholar 

  17. Torlay L, Perrone-Bertolotti M, Thomas E, Baciu M (2017) Machine learning-XGBoost analysis of language networks to classify patients with epilepsy. Brain Inform 4(3):159–169

    CAS  PubMed Central  Article  Google Scholar 

  18. Szczepanek-Parulska E, Wolinski K, Dobruch-Sobczak K et al (2020) S-Detect software vs. EU-TIRADS classification: a dual-center validation of diagnostic performance in differentiation of thyroid nodules. J Clin Med 9(8):2495

    PubMed Central  Article  Google Scholar 

  19. Remonti LR, Kramer CK, Leitão CB, Pinto LC, Gross JL (2015) Thyroid ultrasound features and risk of carcinoma: a systematic review and meta-analysis of observational studies. Thyroid 25(5):538–550

    PubMed Central  Article  Google Scholar 

  20. Rosario PW, Mourão GF (2019) Noninvasive follicular thyroid neoplasm with papillary-like nuclear features (NIFTP): a review for clinicians. Endocr Relat Cancer 26(5):R259–R266

    Article  Google Scholar 

Download references


We thank the ultrasonic physicians who took part in the labeling process of the study, as well as the engineers Yuli She and Yi Jing who gave good suggestions from a standpoint of medical image analysis. Last, we also want to give our sincere thanks to pathological physicians who offered the histopathological results for the thyroid nodules in our study.


This study has received funding from the National Key Research and Development Program of China (No. 2017YFA0700800), the National Natural Science Foundation of China (No. 81871366), the Key Research and Development Program of Shaanxi Province of China (No. 2021SF-346).

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Shaoyi Du or Qi Zhou.

Ethics declarations


The scientific guarantor of this publication is Qi Zhou.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Institutional Review Board approval was obtained.


  • Retrospective

  • Diagnostic study

  • Performed at one institution

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 518 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Jiang, J., Zhang, D. et al. An integrated AI model to improve diagnostic accuracy of ultrasound and output known risk features in suspicious thyroid nodules. Eur Radiol 32, 2120–2129 (2022).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Ultrasound
  • Thyroid cancer
  • Thyroid nodules
  • Artificial intelligence
  • Deep learning