Skip to main content
SpringerLink
Log in

Deep learning to assist composition classification and thyroid solid nodule diagnosis: a multicenter diagnostic study

  • Ultrasound
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

This study aimed to propose a deep learning (DL)–based framework for identifying the composition of thyroid nodules and assessing their malignancy risk.

Methods

We conducted a retrospective multicenter study using ultrasound images from four hospitals. Convolutional neural network (CNN) models were constructed to classify ultrasound images of thyroid nodules into solid and non-solid, as well as benign and malignant. A total of 11,201 images of 6784 nodules were used for training, validation, and testing. The area under the receiver-operating characteristic curve (AUC) was employed as the primary evaluation index.

Results

The models had AUCs higher than 0.91 in the benign and malignant grading of solid thyroid nodules, with the Inception-ResNet AUC being the highest at 0.94. In the test set, the best algorithm for identifying benign and malignant thyroid nodules had a sensitivity of 0.88, and a specificity of 0.86. In the human vs. DL test set, the best algorithm had a sensitivity of 0.93, and a specificity of 0.86. The Inception-ResNet model performed better than the senior physicians (p < 0.001). The sensitivity and specificity of the optimal model based on the external test set were 0.90 and 0.75, respectively.

Conclusions

This research demonstrates that CNNs can assist thyroid nodule diagnosis and reduce the rate of unnecessary fine-needle aspiration (FNA).

Clinical relevance statement

High-resolution ultrasound has led to increased detection of thyroid nodules. This results in unnecessary fine-needle aspiration and anxiety for patients whose nodules are benign. Deep learning can solve these problems to some extent.

Key Points

• Thyroid solid nodules have a high probability of malignancy.

• Our models can improve the differentiation between benign and malignant solid thyroid nodules.

• The differential performance of one model was superior to that of senior radiologists. Applying this could reduce the rate of unnecessary fine-needle aspiration of solid thyroid nodules.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Abbreviations

ACR:

American College of Radiology

AI:

Artificial intelligence

AUC:

Area under the receiver-operating characteristic curve

CEUS:

Contrast-enhanced ultrasound

CI:

Confidence intervals

CNN:

Convolutional neural network

DL:

Deep learning

FNA:

Fine-needle aspiration

Grad-CAM:

Gradient class activation mapping

NPV:

Negative predictive value

PPV:

Positive predictive value

ROC:

Receiver operating characteristic

TI-RADS:

Thyroid Image Radiology and Data System

US:

Ultrasonography

US-FNAB:

Ultrasound-guided fine-needle aspiration biopsy

References

  1. Vaccarella S, Franceschi S, Bray F, Wild CP, Plummer M, Dal Maso L (2016) Worldwide thyroid-cancer epidemic? The increasing impact of overdiagnosis. N Engl J Med 375:614–617

    PubMed  Google Scholar 

  2. Feng R-M, Zong Y-N, Cao S-M, Xu R-H (2019) Current cancer situation in China: good or bad news from the 2018 Global Cancer Statistics? Cancer Commun (Lond) 39:22

    PubMed  Google Scholar 

  3. Ma X, Zhang L (2022) Diagnosis of thyroid nodules based on image enhancement and deep neural networks. Comput Intell Neurosci 2022:5582029

    PubMed  PubMed Central  Google Scholar 

  4. Zhang X, Lee VCS, Rong J, Liu F, Kong H (2022) Multi-channel convolutional neural network architectures for thyroid cancer detection. PLoS One 17:e0262128

    CAS  PubMed  PubMed Central  Google Scholar 

  5. Singh Ospina N, Brito JP, Maraka S et al (2016) Diagnostic accuracy of ultrasound-guided fine needle aspiration biopsy for thyroid malignancy: systematic review and meta-analysis. Endocrine 53:651–661

    CAS  PubMed  Google Scholar 

  6. Yoon JH, Moon HJ, Kim E-K, Kwak JY (2011) Inadequate cytology in thyroid nodules: should we repeat aspiration or follow-up? Ann Surg Oncol 18:1282–1289

    PubMed  Google Scholar 

  7. Chan WK, Sun JH, Liou MJ et al (2021) Using deep convolutional neural networks for enhanced ultrasonographic image diagnosis of differentiated thyroid cancer. Biomedicines 9:1771

    PubMed  PubMed Central  Google Scholar 

  8. Zhang D, Jiang F, Yin R et al (2021) A review of the role of the S-detect computer-aided diagnostic ultrasound system in the evaluation of benign and malignant breast and thyroid masses. Med Sci Monit 27:e931957

    PubMed  PubMed Central  Google Scholar 

  9. Zhao D, Jing Y, Lin X, Zhang B (2021) The value of color Doppler ultrasound in the diagnosis of thyroid nodules: a systematic review and meta-analysis. Gland Surg 10:3369–3377

    PubMed  PubMed Central  Google Scholar 

  10. Zhou Y, Chen H, Qiang J, Wang D (2021) Systematic review and meta-analysis of ultrasonic elastography in the diagnosis of benign and malignant thyroid nodules. Gland Surg 10:2734–2744

    PubMed  PubMed Central  Google Scholar 

  11. Li W, Sun Y, Xu H, Shang W, Dong A (2022) Systematic review and meta-analysis of American College of Radiology TI-RADS inter-reader reliability for risk stratification of thyroid nodules. Front Oncol 12:840516

    PubMed  PubMed Central  Google Scholar 

  12. Nguyen DT, Kang JK, Pham TD, Batchuluun G, Park KR (2020) Ultrasound image-based diagnosis of malignant thyroid nodule using artificial intelligence. Sensors (Basel) 20:1822

    ADS  PubMed  Google Scholar 

  13. Modi L, Sun W, Shafizadeh N et al (2020) Does a higher American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) score forecast an increased risk of malignancy? A correlation study of ACR TI-RADS with FNA cytology in the evaluation of thyroid nodules. Cancer Cytopathol 128:470–481

    PubMed  Google Scholar 

  14. Wang J, Jiang J, Zhang D et al (2022) An integrated AI model to improve diagnostic accuracy of ultrasound and output known risk features in suspicious thyroid nodules. Eur Radiol 32:2120–2129

    PubMed  Google Scholar 

  15. Hoang JK, Middleton WD, Farjat AE et al (2018) Reduction in thyroid nodule biopsies and improved accuracy with American College of Radiology Thyroid Imaging Reporting and Data System. Radiology 287:185–193

    PubMed  Google Scholar 

  16. Sharifi Y, Shafiei S, Tabesh H et al (2022) Observation variation in ultrasonography assessment of thyroid nodules. Asia Ocean J Nucl Med Biol 10:28–35

    PubMed  PubMed Central  Google Scholar 

  17. Russ G, Bonnema SJ, Erdogan MF, Durante C, Ngu R, Leenhardt L (2017) European Thyroid Association guidelines for ultrasound malignancy risk stratification of thyroid nodules in adults: the EU-TIRADS. Eur Thyroid J 6:225–237

    PubMed  PubMed Central  Google Scholar 

  18. Kwak JY, Han KH, Yoon JH et al (2011) Thyroid imaging reporting and data system for US features of nodules: a step in establishing better stratification of cancer risk. Radiology 260:892–899

    PubMed  Google Scholar 

  19. Li W, Zhu Q, Jiang Y et al (2017) Partially cystic thyroid nodules in ultrasound-guided fine needle aspiration: prevalence of thyroid carcinoma and ultrasound features. Medicine (Baltimore) 96:e8689

    PubMed  Google Scholar 

  20. Fang F, Gong Y, Liao L et al (2021) Value of contrast-enhanced ultrasound in partially cystic papillary thyroid carcinomas. Front Endocrinol (Lausanne) 12:783670

    PubMed  Google Scholar 

  21. Brito JP, Gionfriddo MR, Al Nofal A et al (2014) The accuracy of thyroid nodule ultrasound to predict thyroid cancer: systematic review and meta-analysis. J Clin Endocrinol Metab 99:1253–1263

    CAS  PubMed  Google Scholar 

  22. Remonti LR, Kramer CK, Leitão CB, Pinto LC, Gross JL (2015) Thyroid ultrasound features and risk of carcinoma: a systematic review and meta-analysis of observational studies. Thyroid 25:538–550

    PubMed  PubMed Central  Google Scholar 

  23. Hoang JK, Middleton WD, Tessler FN (2021) Update on ACR TI-RADS: successes, challenges, and future directions, from the AJR Special Series on Radiology Reporting and Data Systems. AJR Am J Roentgenol 216:570–578

    PubMed  Google Scholar 

  24. Peng Y, Zhou W, Zhan WW, Xu SY (2017) Ultrasonographic assessment of differential diagnosis between degenerating cystic thyroid nodules and papillary thyroid microcarcinomas. World J Surg 41:2538–2544

    PubMed  Google Scholar 

  25. Peng Q, Niu C, Zhang Q, Zhang M, Chen S, Peng Q (2019) Mummified thyroid nodules: conventional and contrast-enhanced ultrasound features. J Ultrasound Med 38:441–452

    PubMed  Google Scholar 

  26. Chen S, Tang K, Gong Y et al (2022) Value of contrast-enhanced ultrasound in mummified thyroid nodules. Front Endocrinol 13:850698

    Google Scholar 

  27. Ruan J, Xu X, Cai Y et al (2022) A practical CEUS thyroid reporting system for thyroid nodules. Radiology 305:149–159

    PubMed  Google Scholar 

  28. Wu GG, Lv WZ, Yin R et al (2021) Deep learning based on ACR TI-RADS can improve the differential diagnosis of thyroid nodules. Front Oncol 11:575166

    CAS  PubMed  PubMed Central  Google Scholar 

  29. Zhu YC, Jin PF, Bao J, Jiang Q, Wang X (2021) Thyroid ultrasound image classification using a convolutional neural network. Ann Transl Med 9:1526

    PubMed  PubMed Central  Google Scholar 

  30. Chen Y, Gao Z, He Y et al (2022) An artificial intelligence model based on ACR TI-RADS characteristics for US diagnosis of thyroid nodules. Radiology 303:613–619

    PubMed  Google Scholar 

  31. Yao J-C, Wang T, Hou G-H et al (2021) AI detection of mild COVID-19 pneumonia from chest CT scans. Eur Radiol 31:7192–7201

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Wang Y, Guan Q, Lao I et al (2019) Using deep convolutional neural networks for multi-classification of thyroid tumor by histopathology: a large-scale pilot study. Ann Transl Med 7:468

    PubMed  PubMed Central  Google Scholar 

  33. Yu J, Deng Y, Liu T et al (2020) Lymph node metastasis prediction of papillary thyroid carcinoma based on transfer learning radiomics. Nat Commun 11:4807

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  34. Zhao CK, Ren TT, Yin YF et al (2021) A comparative analysis of two machine learning-based diagnostic patterns with thyroid imaging reporting and data system for thyroid nodules: diagnostic performance and unnecessary biopsy rate. Thyroid 31:470–481

    CAS  PubMed  Google Scholar 

  35. Tessler FN, Middleton WD, Grant EG et al (2017) ACR Thyroid Imaging, Reporting and Data System (TI-RADS): white paper of the ACR TI-RADS Committee. J Am Coll Radiol 14:587–595

    PubMed  Google Scholar 

  36. Yunus M, Ahmed Z (2010) Significance of ultrasound features in predicting malignant solid thyroid nodules: need for fine-needle aspiration. J Pak Med Assoc 60:848–853

    PubMed  Google Scholar 

  37. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39:1137–1149

    PubMed  Google Scholar 

  38. Russ G (2016) Risk stratification of thyroid nodules on ultrasonography with the French TI-RADS: description and reflections. Ultrasonography 35:25–38

    PubMed  Google Scholar 

  39. Kim J, Gosnell JE, Roman SA (2020) Geographic influences in the global rise of thyroid cancer. Nat Rev Endocrinol 16:17–29

    PubMed  Google Scholar 

  40. Pitoia F, Miyauchi A (2016) 2015 American Thyroid Association guidelines for thyroid nodules and differentiated thyroid cancer and their implementation in various care settings. Thyroid 26:319–321

    Google Scholar 

  41. Dighe M, Barr R, Bojunga J et al (2017) Thyroid ultrasound: state of the art part 1 - thyroid ultrasound reporting and diffuse thyroid diseases. Med Ultrason 19:79–93

    PubMed  Google Scholar 

  42. Dietrich CF, Müller T, Bojunga J et al (2018) Statement and recommendations on interventional ultrasound as a thyroid diagnostic and treatment procedure. Ultrasound Med Biol 44:14–36

    PubMed  Google Scholar 

  43. Trimboli P, Dietrich CF, David E et al (2018) Ultrasound and ultrasound-related techniques in endocrine diseases. Minerva Endocrinol 43:333–340

    PubMed  Google Scholar 

  44. Hoang JK, Middleton WD, Langer JE et al (2021) Comparison of thyroid risk categorization systems and fine-needle aspiration recommendations in a multi-institutional thyroid ultrasound registry. J Am Coll Radiol 18:1605–1613

    PubMed  Google Scholar 

  45. Qadan L, Ahmed A, Kapila K (2019) Thyroid ultrasound reports: deficiencies and recommendations. Med Princ Pract 28:280–283

    PubMed  Google Scholar 

  46. Buda M, Wildman-Tobriner B, Hoang JK et al (2019) Management of thyroid nodules seen on US images: deep learning may match performance of radiologists. Radiology 292:695–701

    PubMed  Google Scholar 

  47. Tuttle RM, Fagin JA, Minkowitz G et al (2017) Natural history and tumor volume kinetics of papillary thyroid cancers during active surveillance. JAMA Otolaryngol Head Neck Surg 143:1015–1020

    PubMed  PubMed Central  Google Scholar 

  48. Xia J, Chen H, Li Q et al (2017) Ultrasound-based differentiation of malignant and benign thyroid nodules: an extreme learning machine approach. Comput Methods Programs Biomed 147:37–49

    PubMed  Google Scholar 

  49. Ma J, Wu F, Zhu J, Xu D, Kong D (2017) A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics 73:221–230

    PubMed  Google Scholar 

  50. Tessler FN, Middleton WD, Grant EG (2018) Thyroid Imaging Reporting and Data System (TI-RADS): a user’s guide. Radiology 287:1082

    PubMed  Google Scholar 

  51. Gao L, Liu R, Jiang Y et al (2018) Computer-aided system for diagnosing thyroid nodules on ultrasound: a comparison with radiologist-based clinical assessments. Head Neck 40:778–783

    PubMed  Google Scholar 

  52. Li X, Zhang S, Zhang Q et al (2019) Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol 20:193–201

    PubMed  Google Scholar 

  53. Swan KZ, Thomas J, Nielsen VE, Jespersen ML, Bonnema S (2022) External validation of AIBx, an artificial intelligence model for risk stratification, in thyroid nodules. Eur Thyroid J 11:e210129

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors thank ultrasound doctors at the Department of Ultrasound, Zhejiang Cancer Hospital, Shenzhen People’s Hospital, Shanghai Tenth People’s Hospital, and Taizhou Cancer Hospital for data collection. Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

Funding

This work was supported by the National Natural Science Foundation of China (No. 82071946), “Pioneer” and “Leading Goose” R&D Program of Zhejiang (No. 2023C04039), Research Program of National Health Commision Capacity Building and Continuing Education Center (CSJRZC2021JJSJ001), the Natural Science Foundation of Zhejiang Province (No. LZY21F030001), and the Research Program of Zhejiang Provincial Department of Health (No. 2021KY099, 2022KY110, and 2023KY066).

Author information

Author notes

  1. Chen Chen, Yitao Jiang, and Jincao Yao contributed equally to this work.

Authors and Affiliations

  1. Department of Diagnostic Ultrasound Imaging & Interventional Therapy, Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, China

    Chen Chen, Jincao Yao, Yuanzhen Liu, Di Ou, Bojian Feng, Lingyan Zhou & Dong Xu

  2. Wenling Big Data and Artificial Intelligence Institute in Medicine, Taizhou, 317502, China

    Chen Chen, Yuanzhen Liu, Bojian Feng & Dong Xu

  3. Taizhou Key Laboratory of Minimally Invasive Interventional Therapy & Artificial Intelligence, Taizhou Campus of Zhejiang Cancer Hospital (Taizhou Cancer Hospital), Taizhou, 317502, China

    Chen Chen, Yuanzhen Liu, Bojian Feng & Dong Xu

  4. Illuminate, LLC, Shenzhen, Guangdong, 518000, China

    Yitao Jiang

  5. Key Laboratory of Head & Neck Cancer, Translational Research of Zhejiang Province, Hangzhou, 310022, China

    Jincao Yao, Di Ou & Lingyan Zhou

  6. Zhejiang Provincial Research Center for Cancer Intelligent Diagnosis and Molecular Technology, Hangzhou, 310022, China

    Jincao Yao, Di Ou & Lingyan Zhou

  7. The Second Clinical Medical College, Zhejiang Chinese Medical University, Hangzhou, 310053, China

    Min Lai

  8. Department of Ultrasound, Shengzhou People’s Hospital (the First Affiliated Hospital of Zhejiang University Shengzhou Branch), Shengzhou, 312400, China

    Xianping Jiang

  9. Department of Ultrasound, The Second Clinical Medical College, Jinan University, The First Affiliated Hospital, Southern University of Science and Technology, Shenzhen People’s Hospital, Shenzhen, 518020, China

    Jinfeng Xu, Linghu Wu, Yuli Zhou & Fajin Dong

  10. Center of Minimally Invasive Treatment for Tumor, Department of Medical Ultrasound, Shanghai Tenth People’s Hospital, School of Medicine, Tongji University, Shanghai, 200072, China

    Wenwen Yue

Authors
  1. Chen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yitao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jincao Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Min Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuanzhen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xianping Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Di Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Bojian Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lingyan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jinfeng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Linghu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Yuli Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Wenwen Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Fajin Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Dong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Wenwen Yue, Fajin Dong or Dong Xu.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Prof. Dong Xu.

Conflict of Interest

One of the authors (Yitao Jiang) is an employee of Illuminate, LLC, Guangdong, China. The other authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and Biometry

No complex statistical methods were necessary for this paper.

Informed Consent

Written informed consent was waived by the institutional review board.

Ethical Approval

This study was approved by the Ethics Committee of the Cancer Hospital of The University of Chinese Academy of Sciences, Shenzhen People’s Hospital, Shanghai Tenth People’s Hospital, and Taizhou Cancer Hospital.

Study subjects or cohorts overlap

Study subjects or cohorts have not been previously reported.

Methodology

• Retrospective

• Diagnostic study

• Multicenter study

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, C., Jiang, Y., Yao, J. et al. Deep learning to assist composition classification and thyroid solid nodule diagnosis: a multicenter diagnostic study. Eur Radiol 34, 2323–2333 (2024). https://doi.org/10.1007/s00330-023-10269-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-023-10269-z

Keywords

Search

Navigation