Skip to main content
SpringerLink
Log in

Clinical value of artificial intelligence in thyroid ultrasound: a prospective study from the real world

  • Head and Neck
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objective

To evaluate the diagnostic performance of a commercial artificial intelligence (AI)–assisted ultrasonography (US) for thyroid nodules and to validate its value in real-world medical practice.

Materials and methods

From March 2021 to July 2021, 236 consecutive patients with 312 suspicious thyroid nodules were prospectively enrolled in this study. One experienced radiologist performed US examinations with a real-time AI system (S-Detect). US images and AI reports of the nodules were recorded. Nine residents and three senior radiologists were invited to make a “benign” or “malignant” diagnosis based on recorded US images without knowing the AI reports. After referring to AI reports, the diagnosis was made again. The diagnostic performance of AI, residents, and senior radiologists with and without AI reports were analyzed.

Results

The sensitivity, accuracy, and AUC of the AI system were 0.95, 0.84, and 0.753, respectively, and were not statistically different from those of the experienced radiologists, but were superior to those of the residents (all p < 0.01). The AI-assisted resident strategy significantly improved the accuracy and sensitivity for nodules ≤ 1.5 cm (all p < 0.01), while reducing the unnecessary biopsy rate by up to 27.7% for nodules > 1.5 cm (p = 0.034).

Conclusions

The AI system achieved performance, for cancer diagnosis, comparable to that of an average senior thyroid radiologist. The AI-assisted strategy can significantly improve the overall diagnostic performance for less-experienced radiologists, while increasing the discovery of thyroid cancer ≤ 1.5 cm and reducing unnecessary biopsies for nodules > 1.5 cm in real-world medical practice.

Key Points

• The AI system reached a senior radiologist-like level in the evaluation of thyroid cancer and could significantly improve the overall diagnostic performance of residents.

• The AI-assisted strategy significantly improved ≤ 1.5 cm thyroid cancer screening AUC, accuracy, and sensitivity of the residents, leading to an increased detection of thyroid cancer while maintaining a comparable specificity to that of radiologists alone.

• The AI-assisted strategy significantly reduced the unnecessary biopsy rate for thyroid nodules > 1.5 cm by the residents, while maintaining a comparable sensitivity to that of radiologists alone.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

AI:

Artificial intelligence

CNB:

Core needle biopsy

FNA:

Fine-needle aspiration

PTC:

Papillary thyroid carcinoma

US:

Ultrasonography

References

  1. Chmielik E, Rusinek D, Oczko-Wojciechowska M et al (2018) Heterogeneity of thyroid cancer. Pathobiology 85:117–129

    Article  PubMed  Google Scholar 

  2. Sung H, Ferlay J, Siegel RL et al (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 71:209–249

    Article  PubMed  Google Scholar 

  3. Miyauchi A, Ito Y, Oda H (2018) Insights into the management of papillary microcarcinoma of the thyroid. Thyroid 28:23–31

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Kitahara C, Sosa J (2016) The changing incidence of thyroid cancer. Nat Rev Endocrinol 12:646–653

    Article  PubMed  Google Scholar 

  5. Haugen B, Alexander E, Bible K et al (2016) 2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 26:1–133

    Article  PubMed  PubMed Central  Google Scholar 

  6. Kim JH, Baek JH, Lim HK et al (2018) 2017 thyroid radiofrequency ablation guideline: Korean Society of Thyroid Radiology. Korean J Radiol 19:632–655

    Article  PubMed  PubMed Central  Google Scholar 

  7. Gharib H, Papini E, Garber J et al (2016) American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules--2016 update. Endocr Pract 22:622–639

    Article  PubMed  Google Scholar 

  8. Haddad R, Nasr C, Bischoff L et al (2018) NCCN guidelines insights: thyroid carcinoma, version 2.2018. J Natl Compr Cancer Netw 16:1429–1440

    Article  Google Scholar 

  9. Melany M, Chen S (2017) Thyroid cancer: ultrasound imaging and fine-needle aspiration biopsy. Endocrinol Metab Clin N Am 46:691–711

    Article  Google Scholar 

  10. Ozel A, Erturk SM, Ercan A et al (2012) The diagnostic efficiency of ultrasound in characterization for thyroid nodules: how many criteria are required to predict malignancy? Med Ultrason 14:24–28

    PubMed  Google Scholar 

  11. Choi S, Kim E, Kwak J, Kim M, Son E (2010) Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. Thyroid 20:167–172

    Article  PubMed  Google Scholar 

  12. Wienke JR, Chong WK, Fielding JR, Zou KH, Mittelstaedt CA (2003) Sonographic features of benign thyroid nodules: interobserver reliability and overlap with malignancy. J Ultrasound Med 22:1027–1031

    Article  PubMed  Google Scholar 

  13. Kim SH, Park CS, Jung SL et al (2010) Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules. Korean J Radiol 11:149–155

    Article  PubMed  PubMed Central  Google Scholar 

  14. Lee H, Yoon D, Seo Y et al (2018) Intraobserver and interobserver variability in ultrasound measurements of thyroid nodules. J Ultrasound Med 37:173–178

    Article  PubMed  Google Scholar 

  15. Sakorafas GH (2010) Thyroid nodules; interpretation and importance of fine-needle aspiration (FNA) for the clinician - practical considerations. Surg Oncol 19:e130–e139

    Article  PubMed  Google Scholar 

  16. Tuttle RM, Zhang L, Shaha A (2018) A clinical framework to facilitate selection of patients with differentiated thyroid cancer for active surveillance or less aggressive initial surgical management. Expert Rev Endocrinol Metab 13:77–85

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Durante C, Grani G, Lamartina L, Filetti S, Mandel SJ, Cooper DS (2018) The diagnosis and management of thyroid nodules: a review. JAMA 319:914–924

    Article  PubMed  Google Scholar 

  18. Choi Y, Baek J, Park H et al (2017) A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid 27:546–552

    Article  PubMed  Google Scholar 

  19. Yoo Y, Ha E, Cho Y, Kim H, Han M, Kang S (2018) Computer-aided diagnosis of thyroid nodules via ultrasonography: initial clinical experience. Korean J Radiol 19:665–672

    Article  PubMed  PubMed Central  Google Scholar 

  20. Jeong E, Kim H, Ha E, Park S, Cho Y, Han M (2019) Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol 29:1978–1985

    Article  PubMed  Google Scholar 

  21. Chung SR, Baek JH, Lee MK et al (2020) Computer-aided diagnosis system for the evaluation of thyroid nodules on ultrasonography: prospective non-inferiority study according to the experience level of radiologists. Korean J Radiol 21:369–376

    Article  PubMed  PubMed Central  Google Scholar 

  22. Wei Q, Zeng S, Wang L et al (2020) The value of S-Detect in improving the diagnostic performance of radiologists for the differential diagnosis of thyroid nodules. Med Ultrason 22:415–423

    Article  PubMed  Google Scholar 

  23. Cibas ES, Ali SZ (2017) The 2017 Bethesda system for reporting thyroid cytopathology. Thyroid 27:1341–1346

    Article  PubMed  Google Scholar 

  24. Han M, Ha E, Park J (2021) Computer-aided diagnostic system for thyroid nodules on ultrasonography: diagnostic performance based on the thyroid imaging reporting and data system classification and dichotomous outcomes. AJNR Am J Neuroradiol 42:559–565

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Kim H, Ha E, Han M (2019) Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol 45:2672–2678

    Article  PubMed  Google Scholar 

  26. Chambara N, Ying M (2019) The diagnostic efficiency of ultrasound computer-aided diagnosis in differentiating thyroid nodules: a systematic review and narrative synthesis. Cancers (Basel) 11

  27. Zhao WJ, Fu LR, Huang ZM, Zhu JQ, Ma BY (2019) Effectiveness evaluation of computer-aided diagnosis system for the diagnosis of thyroid nodules on ultrasound: a systematic review and meta-analysis. Medicine (Baltimore) 98:e16379

    Article  PubMed  Google Scholar 

  28. Barczyński M, Stopa-Barczyńska M, Wojtczak B, Czarniecka A, Konturek A (2020) Clinical validation of S-Detect mode in semi-automated ultrasound classification of thyroid lesions in surgical office. Gland Surg 9:S77–S85

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This study is supported by Beijing Municipal Science & Technology Commission (No. Z221100003522001).

Author information

Authors and Affiliations

  1. Medical School of Chinese PLA, No. 28 Fuxing Rd, Haidian District, Beijing, 100853, China

    Yingying Li, Yihao Liu & Zhen Yang

  2. Department of Ultrasound, The First Medical Center of Chinese PLA General Hospital, No. 28 Fuxing Rd, Haidian District, Beijing, 100853, China

    Yingying Li, Jing Xiao, Lin Yan, Xinyang Li, Mingbo Zhang & Yukun Luo

Authors
  1. Yingying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yihao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xinyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Mingbo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yukun Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Mingbo Zhang or Yukun Luo.

Ethics declarations

Guarantor

The scientific guarantors of this publication are Mingbo Zhang and Yukun Luo.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Ethical approval was obtained from the Institutional Ethics Committee of the Chinese PLA General Hospital.

Methodology

• prospective

• diagnostic or prognostic study

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 57 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Liu, Y., Xiao, J. et al. Clinical value of artificial intelligence in thyroid ultrasound: a prospective study from the real world. Eur Radiol 33, 4513–4523 (2023). https://doi.org/10.1007/s00330-022-09378-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-09378-y

Keywords

Search

Navigation