Skip to main content

Advertisement

SpringerLink
Log in

A comparison of artificial intelligence versus radiologists in the diagnosis of thyroid nodules using ultrasonography: a systematic review and meta-analysis

  • Head and Neck
  • Published:
European Archives of Oto-Rhino-Laryngology Aims and scope Submit manuscript

Abstract

Background

Thyroid nodules are common. Ultrasonography (US) is the first investigation for thyroid nodules. Artificial Intelligence (AI) is widely integrated into medical diagnosis to provide additional information. The primary objective of this study was to accumulate the pooled sensitivity and specificity between all available AI and radiologists using thyroid US imaging. The secondary objective was to compare AI’s diagnostic performance to that of radiologists.

Materials and methods

A systematic review meta-analysis. PubMed, Scopus, Web of Science, and Cochrane Library data were searched for studies from inception until June 11, 2020.

Results

Twenty five studies were included in this meta-analysis. The pooled sensitivity and specificity of AI were 0.86 (95% CI 0.81–0.91) and 0.78 (95% CI 0.73–0.83), respectively. The pooled sensitivity and specificity of radiologists were 0.85 (95% CI 0.80–0.89) and 0.82 (95% CI 0.77–0.86), respectively. The accuracy of AI and radiologists is equivalent in terms of AUC [AI 0.89 (95% CI 0.86–0.92), radiologist 0.91 (95% CI 0.88–0.93)]. The diagnostic odd ratio (DOR) between AI 23.10 (95% CI 14.20–37.58) and radiologists 27.12 (95% CI 17.45–42.16) had no statistically significant difference (P = 0.56). Meta-regression analysis revealed that Deep Learning AI had significantly greater sensitivity and specificity than classic machine learning AI (P < 0.001).

Conclusion

AI demonstrated comparable performance to radiologists in diagnosing benign and malignant thyroid nodules using ultrasonography. Additional research to establish its equivalency should be conducted.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Dean DS, Gharib H (2008) Epidemiology of thyroid nodules. Best Pract Res Clin Endocrinol Metab 22:901–911. https://doi.org/10.1016/j.beem.2008.09.019

    Article  PubMed  Google Scholar 

  2. Tan GH, Gharib H (1997) Thyroid incidentalomas: management approaches to nonpalpable nodules discovered incidentally on thyroid imaging. Ann Intern Med 126:226–231

    Article  CAS  Google Scholar 

  3. Tangjaturonrasme N, Vatanasapt P, Bychkov A (2018) Epidemiology of head and neck cancer in Thailand. Asia Pac J Clin Oncol 14:16–22. https://doi.org/10.1111/ajco.12757

    Article  PubMed  Google Scholar 

  4. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer (2016) The American thyroid association guidelines task force on thyroid nodules and differentiated thyroid cancer. 26:1–133. https://doi.org/10.1089/thy.2015.0020

  5. Park CS, Kim SH, Jung SL, Kang BJ, Kim JY, Choi JJ et al (2010) Observer variability in the sonographic evaluation of thyroid nodules. J Clin Ultrasound 38:287–293. https://doi.org/10.1002/jcu.20689

    Article  PubMed  Google Scholar 

  6. Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB et al (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155:529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009

    Article  PubMed  Google Scholar 

  7. Deeks JJ, Macaskill P, Irwig L (2005) The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 58:882–893

    Article  Google Scholar 

  8. Hartzes AM, Morgan CJ (2019) Meta-analysis for diagnostic tests. J Nucl Cardiol 26:68–71

    Article  Google Scholar 

  9. Kim KW, Lee J, Choi SH, Huh J, Park SH (2015) Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-part I. General guidance and tips. Korean J Radiol 16:1175–1187. https://doi.org/10.3348/kjr.2015.16.6.1188

    Article  PubMed  PubMed Central  Google Scholar 

  10. Lee J, Kim KW, Choi SH, Huh J, Park SH (2015) Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-part II. Statistical methods of meta-analysis. Korean J Radiol 16:1188–1196. https://doi.org/10.3348/kjr.2015.16.6.1188

    Article  PubMed  PubMed Central  Google Scholar 

  11. Chambara N, Ying M (2019) The diagnostic efficiency of ultrasound computer-aided diagnosis in differentiating thyroid nodules: a systematic review and narrative synthesis. Cancers (Basel). https://doi.org/10.3390/cancers11111759

    Article  Google Scholar 

  12. Xu L, Gao J, Wang Q, Yin J, Yu P, Bai B et al (2020) Computer-aided diagnosis systems in diagnosing malignant thyroid nodules on ultrasonography: a systematic review and meta-analysis. Eur Thyroid J 9:186–193. https://doi.org/10.1159/000504390

    Article  PubMed  Google Scholar 

  13. Zhao WJ, Fu LR, Huang ZM, Zhu JQ, Ma BY, Tarantino G (2019) Effectiveness evaluation of computer-aided diagnosis system for the diagnosis of thyroid nodules on ultrasound: a systematic review and meta-analysis. Medicine (United States). https://doi.org/10.1097/MD.0000000000016379

    Article  PubMed  PubMed Central  Google Scholar 

  14. Moon WJ, Baek JH, Jung SL, Kim DW, Kim EK, Kim JY et al (2011) Ultrasonography and the ultrasound-based management of thyroid nodules: consensus statement and recommendations. Korean J Radiol 12:1–14. https://doi.org/10.3348/kjr.2011.12.1.1

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kim EK, Park CS, Chung WY, Oh KK, Kim DI, Lee JT et al (2002) New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol 178:687–691. https://doi.org/10.2214/ajr.178.3.1780687

    Article  PubMed  Google Scholar 

  16. Barczyński M, Stopa-Barczyńska M, Wojtczak B, Czarniecka A, Konturek A (2020) Clinical validation of S-Detect(TM) mode in semi-automated ultrasound classification of thyroid lesions in surgical office. Gland Surg 9:S77–S85. https://doi.org/10.21037/gs.2019.12.23

    Article  PubMed  PubMed Central  Google Scholar 

  17. Chung SR, Baek JH, Lee MK, Ahn Y, Choi YJ, Sung TY et al (2020) Computer-aided diagnosis system for the evaluation of thyroid nodules on ultrasonography: prospective non-inferiority study according to the experience level of radiologists. Korean J Radiol 21:369–376. https://doi.org/10.3348/kjr.2019.0581

    Article  PubMed  PubMed Central  Google Scholar 

  18. Jin Z, Zhu Y, Zhang S, Xie F, Zhang M, Zhang Y et al (2020) Ultrasound computer-aided diagnosis (CAD) based on the thyroid imaging reporting and data system (TI-RADS) to distinguish benign from malignant thyroid nodules and the diagnostic performance of radiologists with different diagnostic experience. Med Sci Monit. https://doi.org/10.12659/MSM.918452

    Article  PubMed  PubMed Central  Google Scholar 

  19. Molnár K, Kálmán E, Hári Z, Giyab O, Gáspár T, Rucz K et al (2020) False-positive malignant diagnosis of nodule mimicking lesions by computer-aided thyroid nodule analysis in clinical ultrasonography practice. Diagnostics (Basel). https://doi.org/10.3390/diagnostics10060378

    Article  Google Scholar 

  20. Shin I, Kim YJ, Han K, Lee E, Kim HJ, Shin JH et al (2020) Application of machine learning to ultrasound images to differentiate follicular neoplasms of the thyroid gland. Ultrasonography 39:257–265. https://doi.org/10.14366/usg.19069

    Article  PubMed  PubMed Central  Google Scholar 

  21. Sun C, Zhang Y, Chang Q, Liu T, Zhang S, Wang X et al (2020) Evaluation of a deep learning-based computer-aided diagnosis system for distinguishing benign from malignant thyroid nodules in ultrasound images. Med Phys 47:3952–3960. https://doi.org/10.1002/mp.14301

    Article  PubMed  Google Scholar 

  22. Zhou H, Jin Y, Dai L, Zhang M, Qiu Y, Wang K et al (2020) Differential diagnosis of benign and malignant thyroid nodules using deep learning radiomics of thyroid ultrasound images. Eur J Radiol 127:108992. https://doi.org/10.1016/j.ejrad.2020.108992

    Article  PubMed  Google Scholar 

  23. Buda M, Wildman-Tobriner B, Hoang JK, Thayer D, Tessler FN, Middleton WD et al (2019) Management of thyroid nodules seen on us images: deep learning may match performance of radiologists. Radiology 292:695–701. https://doi.org/10.1148/radiol.2019181343

    Article  PubMed  Google Scholar 

  24. Fresilli D, Grani G, De Pascali ML, Alagna G, Tassone E, Ramundo V et al (2020) Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners. J Ultrasound 23:169–174. https://doi.org/10.1007/s40477-020-00453-y

    Article  PubMed  PubMed Central  Google Scholar 

  25. Jeong EY, Kim HL, Ha EJ, Park SY, Cho YJ, Han M (2019) Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol 29:1978–1985. https://doi.org/10.1007/s00330-018-5772-9

    Article  PubMed  Google Scholar 

  26. Kim HL, Ha EJ, Han M (2019) Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol 45:2672–2678. https://doi.org/10.1016/j.ultrasmedbio.2019.05.032

    Article  PubMed  Google Scholar 

  27. Lee E, Ha H, Kim HJ, Moon HJ, Byon JH, Huh S et al (2019) Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks. Sci Rep 9:19854. https://doi.org/10.1038/s41598-019-56395-x

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Li X, Zhang S, Zhang Q, Wei X, Pan Y, Zhao J et al (2019) Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol 20:193–201. https://doi.org/10.1016/s1470-2045(18)30762-9

    Article  PubMed  Google Scholar 

  29. Liu T, Guo Q, Lian C, Ren X, Liang S, Yu J et al (2019) Automated detection and classification of thyroid nodules in ultrasound images using clinical-knowledge-guided convolutional neural networks. Med Image Anal 58:101555. https://doi.org/10.1016/j.media.2019.101555

    Article  PubMed  Google Scholar 

  30. Park VY, Han K, Seong YK, Park MH, Kim EK, Moon HJ et al (2019) Diagnosis of thyroid nodules: performance of a deep learning convolutional neural network model vs radiologists. Sci Rep 9:17843. https://doi.org/10.1038/s41598-019-54434-1

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Reverter JL, Vázquez F, Puig-Domingo M (2019) Diagnostic performance evaluation of a computer-assisted imaging analysis system for ultrasound risk stratification of thyroid nodules. AJR Am J Roentgenol 213:169–174. https://doi.org/10.2214/ajr.18.20740

    Article  PubMed  Google Scholar 

  32. Wang L, Yang S, Yang S, Zhao C, Tian G, Gao Y et al (2019) Automatic thyroid nodule recognition and diagnosis in ultrasound imaging with the YOLOv2 neural network. World J Surg Oncol 17:12. https://doi.org/10.1186/s12957-019-1558-z

    Article  PubMed  PubMed Central  Google Scholar 

  33. Xia S, Yao J, Zhou W, Dong Y, Xu S, Zhou J et al (2019) A computer-aided diagnosing system in the evaluation of thyroid nodules-experience in a specialized thyroid center. World J Surg Oncol 17:210. https://doi.org/10.1186/s12957-019-1752-z

    Article  PubMed  PubMed Central  Google Scholar 

  34. Zhang B, Tian J, Pei S, Chen Y, He X, Dong Y et al (2019) Machine learning-assisted system for thyroid nodule diagnosis. Thyroid 29:858–867. https://doi.org/10.1089/thy.2018.0380

    Article  PubMed  Google Scholar 

  35. Gao L, Liu R, Jiang Y, Song W, Wang Y, Liu J et al (2018) Computer-aided system for diagnosing thyroid nodules on ultrasound: a comparison with radiologist-based clinical assessments. Head Neck 40:778–783. https://doi.org/10.1002/hed.25049

    Article  PubMed  Google Scholar 

  36. Gitto S, Grassi G, De Angelis C, Monaco CG, Sdao S, Sardanelli F et al (2019) A computer-aided diagnosis system for the assessment and characterization of low-to-high suspicion thyroid nodules on ultrasound. Radiol Med 124:118–125. https://doi.org/10.1007/s11547-018-0942-z

    Article  PubMed  Google Scholar 

  37. Yoo YJ, Ha EJ, Cho YJ, Kim HL, Han M, Kang SY (2018) Computer-aided diagnosis of thyroid nodules via ultrasonography: initial clinical experience. Korean J Radiol 19:665–672. https://doi.org/10.3348/kjr.2018.19.4.665

    Article  PubMed  PubMed Central  Google Scholar 

  38. Choi YJ, Baek JH, Park HS, Shim WH, Kim TY, Shong YK et al (2017) A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid 27:546–552. https://doi.org/10.1089/thy.2016.0372

    Article  PubMed  Google Scholar 

  39. Ko SY, Lee JH, Yoon JH, Na H, Hong E, Han K et al (2019) Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head Neck 41:885–891. https://doi.org/10.1002/hed.25415

    Article  PubMed  Google Scholar 

  40. Wu H, Deng Z, Zhang B, Liu Q, Chen J (2016) Classifier model based on machine learning algorithms: application to differential diagnosis of suspicious thyroid nodules via sonography. AJR Am J Roentgenol 207:859–864. https://doi.org/10.2214/ajr.15.15813

    Article  PubMed  Google Scholar 

Download references

Funding

No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. Department of Otolaryngology, Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand

    Pimrada Potipimpanon, Natamon Charakorn & Prakobkiat Hirunwiwatkul

Authors
  1. Pimrada Potipimpanon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Natamon Charakorn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Prakobkiat Hirunwiwatkul
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pimrada Potipimpanon.

Ethics declarations

Conflict of interest

No conflict of interest to be disclosed. The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

Research does not involve human participants and/or animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PNG 251 KB)

Supplementary file2 (PNG 252 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Potipimpanon, P., Charakorn, N. & Hirunwiwatkul, P. A comparison of artificial intelligence versus radiologists in the diagnosis of thyroid nodules using ultrasonography: a systematic review and meta-analysis. Eur Arch Otorhinolaryngol 279, 5363–5373 (2022). https://doi.org/10.1007/s00405-022-07436-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00405-022-07436-1

Keywords

Search

Navigation