Skip to main content
SpringerLink
Log in

Development of a machine learning-based fine-grained risk stratification system for thyroid nodules using predefined clinicoradiological features

  • Ultrasound
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objective

We constructed and validated a machine learning-based malignancy risk estimation model using predefined clinicoradiological features, and evaluated its clinical utility for the management of thyroid nodules.

Methods

In total, 5708 benign (n = 4597) and malignant (n = 1111) thyroid nodules were collected from 5081 consecutive patients treated in 26 institutions. Seventeen experienced radiologists evaluated nodule characteristics on ultrasonographic images. Eight predictive models were used to stratify the thyroid nodules according to malignancy risk; model performance was assessed via nested 10-fold cross-validation. The best-performing algorithm was externally validated using data for 454 thyroid nodules from a tertiary hospital, then compared to the Thyroid Imaging Reporting and Data System (TIRADS)–based interpretations of radiologists (American College of Radiology, European and Korean TIRADS, and AACE/ACE/AME guidelines).

Results

The area under the receiver operating characteristic (AUROC) curves of the algorithms ranged from 0.773 to 0.862. The sensitivities, specificities, positive predictive values, and negative predictive values of the best-performing models were 74.1–76.6%, 80.9–83.4%, 49.2–51.9%, and 93.0–93.5%, respectively. For the external validation set, the ElasticNet values were 83.2%, 89.2%, 81.8%, and 90.1%, respectively. The corresponding TIRADS values were 66.5–85.0%, 61.3–80.8%, 45.9–72.1%, and 81.5–90.3%, respectively. The new model exhibited a significantly higher AUROC and specificity than did the TIRADS risk stratification, although its sensitivity was similar.

Conclusion

We developed a reliable machine learning–based predictive model that demonstrated enhanced specificity when stratifying thyroid nodules according to malignancy risk. This system will contribute to improved personalized management of thyroid nodules.

Key Points

• The area under the receiver operating characteristic (AUROC) curve, sensitivity, and specificity of our model were 0.914, 83.2%, and 89.2%, respectively (derived using the validation dataset).

• Compared to the TIRADS values, the AUROC and specificity are significantly higher, while the sensitivity is similar.

• An interactive version of our AI algorithm is at http://tirads.cdss.co.kr .

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

AI:

Artificial intelligence

AUROC:

Area under the receiver operating characteristic

CI:

Confidence interval

FNA:

Fine-needle aspiration

LASSO:

Least Absolute Shrinkage and Selection Operator

ML:

Machine learning

NPV:

Negative predictive value

PPV:

Positive predictive value

RF:

Random forest

RSS:

Risk stratification systems

TIRADS:

Thyroid Imaging Reporting and Data System

US:

Ultrasonography

XGBoost:

Extreme gradient boosting

References

  1. Ha EJ, Chung SR, Na DG et al (2021) 2021 Korean Thyroid Imaging Reporting and Data System and Imaging-Based Management of Thyroid Nodules: Korean Society of Thyroid Radiology Consensus Statement and Recommendations. Korean J Radiol 22:2094–2123

    Article  PubMed  PubMed Central  Google Scholar 

  2. Tessler FN, Middleton WD, Grant EG et al (2017) ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol 14:587–595

    Article  PubMed  Google Scholar 

  3. Gharib H, Papini E, Garber JR et al (2016) American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi Medical Guidelines for Clinical Practice for the Diagnosis and Management of Thyroid Nodules--2016 Update. Endocr Pract 22:622–639

    Article  PubMed  Google Scholar 

  4. Russ G, Bonnema SJ, Erdogan MF, Durante C, Ngu R, Leenhardt L (2017) European Thyroid Association Guidelines for Ultrasound Malignancy Risk Stratification of Thyroid Nodules in Adults: The EU-TIRADS. Eur Thyroid J 6:225–237

    Article  PubMed  PubMed Central  Google Scholar 

  5. Ha EJ, Baek JH (2021) Applications of machine learning and deep learning to thyroid imaging: where do we stand? Ultrasonography 40:23–29

    Article  PubMed  Google Scholar 

  6. Ha EJ, Baek JH, Na DG (2017) Risk stratification of thyroid nodules on ultrasonography: current status and perspectives. Thyroid 27:1463–1468

    Article  PubMed  Google Scholar 

  7. Kwak JY, Jung I, Baek JH et al (2013) Image reporting and characterization system for ultrasound features of thyroid nodules: multicentric Korean retrospective study. Korean J Radiol 14:110–117

    Article  PubMed  Google Scholar 

  8. Choi YJ, Baek JH, Baek SH et al (2015) Web-based malignancy risk estimation for thyroid nodules using ultrasonography characteristics: development and validation of a predictive model. Thyroid 25:1306–1312

    Article  PubMed  Google Scholar 

  9. Li LR, Du B, Liu HQ, Chen C (2020) Artificial intelligence for personalized medicine in thyroid cancer: current status and future perspectives. Front Oncol 10:604051

    Article  PubMed  Google Scholar 

  10. Peng S, Liu Y, Lv W et al (2021) Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic study. Lancet Digit Health 3:e250–e259

    Article  CAS  PubMed  Google Scholar 

  11. Han M, Ha EJ, Park JH (2021) Computer-aided diagnostic system for thyroid nodules on ultrasonography: diagnostic performance based on the thyroid imaging reporting and data system classification and dichotomous outcomes. AJNR Am J Neuroradiol 42:559–565

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Kim HL, Ha EJ, Han M (2019) Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol 45:2672–2678

    Article  PubMed  Google Scholar 

  13. Yoo YJ, Ha EJ, Cho YJ, Kim HL, Han M, Kang SY (2018) Computer-aided diagnosis of thyroid nodules via ultrasonography: initial clinical experience. Korean J Radiol 19:665–672

    Article  PubMed  PubMed Central  Google Scholar 

  14. Nguyen DT, Kang JK, Pham TD, Batchuluun G, Park KR (2020) Ultrasound image-based diagnosis of malignant thyroid nodule using artificial intelligence. Sensors (Basel) 20:1822

    Article  PubMed  Google Scholar 

  15. Thomas J, Haertling T (2020) AIBx, artificial intelligence model to risk stratify thyroid nodules. Thyroid 30:878–884

    Article  PubMed  Google Scholar 

  16. Thomas J, Ledger GA, Mamillapalli CK (2020) Use of artificial intelligence and machine learning for estimating malignancy risk of thyroid nodules. Curr Opin Endocrinol Diabetes Obes 27:345–350

    Article  PubMed  Google Scholar 

  17. Ha EJ, Shin JH, Na DG et al (2021) Comparison of the diagnostic performance of the modified Korean Thyroid Imaging Reporting and Data System for thyroid malignancy with three international guidelines. Ultrasonography 40:594–601

    Article  PubMed  PubMed Central  Google Scholar 

  18. Chung SR, Ahn HS, Choi YJ et al (2021) Diagnostic performance of the Modified Korean Thyroid Imaging Reporting and Data System for Thyroid Malignancy: a multicenter validation study. Korean J Radiol 22:1579–1586

    Article  PubMed  PubMed Central  Google Scholar 

  19. Robin X, Turck N, Hainard A et al (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77

    Article  PubMed  PubMed Central  Google Scholar 

  20. O'Brien R, Ishwaran H (2019) A random forests quantile classifier for class imbalanced data. Pattern Recognit 90:232–249

    Article  PubMed  PubMed Central  Google Scholar 

  21. Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33:1–22

    Article  PubMed  PubMed Central  Google Scholar 

  22. Tianqi Chen CG (2016) Xgboost: a scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data

  23. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F, Chang CC et al (2014) e1071: Misc Functions of the Department of Statistics (e1071), TU Wien. R package version 13

  24. Kim EK, Park CS, Chung WY et al (2002) New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol 178:687–691

    Article  PubMed  Google Scholar 

  25. Middleton WD, Teefey SA, Reading CC et al (2017) Multiinstitutional analysis of thyroid nodule risk stratification using the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am J Roentgenol 208:1331–1341

    Article  PubMed  Google Scholar 

  26. Chang Y, Paul AK, Kim N et al (2016) Computer-aided diagnosis for classifying benign versus malignant thyroid nodules based on ultrasound images: a comparison with radiologist-based assessments. Med Phys 43:554

    Article  PubMed  Google Scholar 

  27. Wu G, Lv W, Yin R et al (2021) Deep learning based on ACR TI-RADS can improve the differential diagnosis of thyroid nodules. Front Oncol 11:575166

    Article  PubMed  PubMed Central  Google Scholar 

  28. Zhang S, Du H, Jin Z et al (2020) A novel interpretable computer-aided diagnosis system of thyroid nodules on ultrasound based on clinical experience. IEEE Access 8:53223–53231

    Article  Google Scholar 

  29. Jin Z, Zhu Y, Zhang S et al (2021) Diagnosis of thyroid cancer using a TI-RADS-based computer-aided diagnosis system: a multicenter retrospective study. Clin Imaging 30:43–49

    Article  Google Scholar 

Download references

Funding

The authors state that this work was supported by the National Research Foundation of Korea (NRF) grant by the Korea government (MSIT) (#2021R1C1C100698711), by the 2018 intramural research fund of Ajou University Medical Center, and by grant No 03-2021-2230 from the SNUH Research Fund.

Author information

Authors and Affiliations

  1. Department of Radiology, Ajou University School of Medicine, Wonchon-Dong, Yeongtong-Gu, Suwon, 16499, South Korea

    Eun Ju Ha, Jeong Hoon Lee & Da Hyun Lee

  2. Department of Radiology, GangNeung Asan Hospital, University of Ulsan College of Medicine, Gangneung-si, Gangwon-do, 25440, South Korea

    Dong Gyu Na

  3. Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, South Korea

    Ji-hoon Kim

Authors
  1. Eun Ju Ha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeong Hoon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Da Hyun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dong Gyu Na
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ji-hoon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ji-hoon Kim.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Ji-hoon Kim.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

This study was approved by our institutional review board.

Methodology

  • retrospective

  • multicenter cohort study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 36 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ha, E.J., Lee, J.H., Lee, D.H. et al. Development of a machine learning-based fine-grained risk stratification system for thyroid nodules using predefined clinicoradiological features. Eur Radiol 33, 3211–3221 (2023). https://doi.org/10.1007/s00330-022-09376-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-09376-0

Keywords

Search

Navigation