Skip to main content

Advertisement

SpringerLink
Log in

Comparison of diagnostic accuracy and utility of artificial intelligence–optimized ACR TI-RADS and original ACR TI-RADS: a multi-center validation study based on 2061 thyroid nodules

  • Head and Neck
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objective

To determine if artificial intelligence–based modification of the Thyroid Imaging Reporting Data System (TI-RADS) would be better than the current American College of Radiology (ACR) TI-RADS for risk stratification of thyroid nodules.

Methods

A total of 2061 thyroid nodules (in 1859 patients) sampled with fine-needle aspiration or operation were retrospectively analyzed between January 2017 and July 2020. Two radiologists blinded to the pathologic diagnosis evaluated nodule features in five ultrasound categories and assigned TI-RADS scores by both ACR TI-RADS and AI TI-RADS. Inter-rater agreement was assessed by asking another two radiologists to score a set of 100 nodules independently. The reference standard was postoperative pathological or cytopathological diagnosis according to the Bethesda system. Inter-rater agreement was determined using intraclass correlation coefficient (ICC).

Results

AI TI-RADS assigned lower TI-RADS risk levels than ACR TI-RADS (p < 0.001) and had larger area under receiver operating characteristic curve (0.762 vs. 0.679, p < 0.001). The sensitivities of ACR TI-RADS and AI TI-RADS were similar (86.7% vs. 82.2%, p = 0.052), but specificity was higher with AI TI-RADS (70.2% vs. 49.2%, p < 0.001). AI TI-RADS downgraded 743 (48.63%) benign nodules, indicating that 328 (42.3% of 776 biopsied nodules) unnecessary fine-needle aspirations (FNA) could have been avoided. Inter-rater agreement was better with AI TI-RADS than with ACR TI-RADS (ICC, 0.808 vs. 0.861, p < 0.001).

Conclusion

AI TI-RADS can achieve meaningful reduction in the number of benign thyroid nodules recommended for biopsy and significantly improve specificity despite a slight decrease in sensitivity.

Key Points

• AI TI-RADS assigned lower TI-RADS risk levels than ACR TI-RADS, showing similar sensitivity but higher specificity.

• Half of the benign nodules can be downgraded of which 42.3% of biopsy nodules avoided unnecessary fine-needle aspiration (FNA).

• AI TI-RADS had a better overall inter-rater agreement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

ACR:

American College of Radiology

AI:

Artificial intelligence

AUC:

Area under receiver operating characteristic curve

CI:

Confidence interval

FNA:

Fine-needle aspiration

ICC:

Interclass correlation coefficient

ROC:

Receiver operating characteristic curve

SD:

Standard deviation

TI-RADS:

Thyroid Imaging Reporting and Data System

US:

Ultrasound

References

  1. Wiltshire J, Drake T, Uttley L, Balasubramanian S (2016) Systematic review of trends in the incidence rates of thyroid cancer. Thyroid 26:1541–1552

  2. Vaccarella S, Franceschi S, Bray F, Wild C, Plummer M, Dal Maso L (2016) Worldwide thyroid-cancer epidemic? The increasing impact of overdiagnosis. N Engl J Med 375:614–617

  3. Li Y, Teng D, Ba J et al (2020) Efficacy and safety of long-term universal salt iodization on thyroid disorders: epidemiological evidence from 31 provinces of mainland China. Thyroid 30:568–579

    Article  CAS  Google Scholar 

  4. Singh Ospina N, Brito J, Maraka S et al (2016) Diagnostic accuracy of ultrasound-guided fine needle aspiration biopsy for thyroid malignancy: systematic review and meta-analysis. Endocrine 53:651–661

    Article  CAS  Google Scholar 

  5. Hoang JK, Middleton WD, Farjat AE et al (2018) Interobserver variability of sonographic features used in the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am J Roentgenol 211:162–167

    Article  Google Scholar 

  6. Smith-Bindman R, Lebda P, Feldstein V et al (2013) Risk of thyroid cancer based on thyroid ultrasound imaging characteristics: results of a population-based study. JAMA Intern Med 173:1788–1796

  7. Ahn H, Kim H, Welch H (2014) Korea’s thyroid-cancer “epidemic”--screening and overdiagnosis. N Engl J Med 371:1765–1767

    Article  Google Scholar 

  8. Srinivas M, Amogh V, Gautam M et al (2016) A prospective study to evaluate the reliability of Thyroid Imaging Reporting and Data System in differentiation between benign and malignant thyroid lesions. J Clin Imaging Sci 6:5

  9. Horvath E, Majlis S, Rossi R et al (2009) An ultrasonogram reporting system for thyroid nodules stratifying cancer risk for clinical management. J Clin Endocrinol Metab 94:1748–1751

  10. Tessler F, Middleton W, Grant E et al (2017) ACR Thyroid Imaging, Reporting and Data System (TI-RADS): white paper of the ACR TI-RADS Committee. J Am Coll Radiol 14:587–595

  11. Wildman-Tobriner B, Buda M, Hoang J et al (2019) Using artificial intelligence to revise ACR TI-RADS risk stratification of thyroid nodules: diagnostic accuracy and utility. Radiology 292:112–119

    Article  Google Scholar 

  12. Hoang J, Middleton W, Farjat A et al (2018) Reduction in thyroid nodule biopsies and improved accuracy with American College of Radiology Thyroid Imaging Reporting and Data System. Radiology 287:185–193

    Article  Google Scholar 

  13. Pandya A, Caoili E, Jawad-Makki F et al (2020) Retrospective cohort study of 1947 thyroid nodules: a comparison of the 2017 American College of Radiology TI-RADS and the 2015 American Thyroid Association Classifications. AJR Am J Roentgenol 214:900–906

  14. Tan L, Tan Y, Tan S (2020) Diagnostic accuracy and ability to reduce unnecessary FNAC: a comparison between four Thyroid Imaging Reporting Data System (TI-RADS) versions. Clin Imaging 65:133–137

  15. Cibas E, Ali S (2017) The 2017 Bethesda system for reporting thyroid cytopathology. Thyroid 27:1341–1346

    Article  Google Scholar 

  16. Middleton W, Teefey S, Reading C et al (2018) Comparison of performance characteristics of American College of Radiology TI-RADS, Korean Society of Thyroid Radiology TIRADS, and American Thyroid Association Guidelines. AJR Am J Roentgenol 210:1148–1154

    Article  Google Scholar 

  17. Lauria Pantano A, Maddaloni E, Briganti S et al (2018) Differences between ATA, AACE/ACE/AME and ACR TI-RADS ultrasound classifications performance in identifying cytological high-risk thyroid nodules. Eur J Endocrinol 178:595–603

  18. Yoon J, Lee H, Kim E, Moon H, Kwak J (2016) Malignancy risk stratification of thyroid nodules: comparison between the Thyroid Imaging Reporting and Data System and the 2014 American Thyroid Association Management Guidelines. Radiology 278:917–924

    Article  Google Scholar 

  19. Kim D, Chung S, Choi S, Kim K (2020) Accuracy of thyroid imaging reporting and data system category 4 or 5 for diagnosing malignancy: a systematic review and meta-analysis. Eur Radiol 30:5611–5624

  20. Ha SM, Baek JH, Na DG et al (2019) Diagnostic performance of practice guidelines for thyroid nodules: thyroid nodule size versus biopsy rates. Radiology 291:92–99

    Article  Google Scholar 

  21. Grani G, Lamartina L, Ascoli V et al (2019) Reducing the number of unnecessary thyroid biopsies while improving diagnostic accuracy: toward the “right” TIRADS. J Clin Endocrinol Metab 104:95–102

    Article  Google Scholar 

  22. La Vecchia C, Malvezzi M, Bosetti C et al (2015) Thyroid cancer mortality and incidence: a global overview. Int J Cancer 136:2187–2195

  23. Lee M, Kim E, Kwak J, Kim M (2009) Partially cystic thyroid nodules on ultrasound: probability of malignancy and sonographic differentiation. Thyroid 19:341–346

    Article  Google Scholar 

  24. Frates M, Benson C, Doubilet P et al (2006) Prevalence and distribution of carcinoma in patients with solitary and multiple thyroid nodules on sonography. J Clin Endocrinol Metab 91:3411–3417

    Article  CAS  Google Scholar 

  25. Bellantone R, Lombardi C, Raffaelli M et al (2004) Management of cystic or predominantly cystic thyroid nodules: the role of ultrasound-guided fine-needle aspiration biopsy. Thyroid 14:43–47

    Article  Google Scholar 

  26. Shi Y, Jin Y, Zheng L (2020) Partially cystic thyroid nodules on ultrasound: the associated factors for malignancy. Clin Hemorheol Microcirc 74:373–381

  27. Kim D, Park J, In H, Choo H, Ryu J, Jung S (2012) Ultrasound-based diagnostic classification for solid and partially cystic thyroid nodules. AJNR Am J Neuroradiol 33:1144–1149

  28. Kim D, Lee E, In H, Kim S (2010) Sonographic differentiation of partially cystic thyroid nodules: a prospective study. AJNR Am J Neuroradiol 31:1961–1966

    Article  CAS  Google Scholar 

  29. Middleton W, Teefey S, Reading C et al (2017) Multiinstitutional analysis of thyroid nodule risk stratification using the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am J Roentgenol 208:1331–1341

    Article  Google Scholar 

  30. Lee Y, Kim D, In H et al (2011) Differentiation between benign and malignant solid thyroid nodules using an US classification system. Korean J Radiology 12:559–567

  31. Choi Y, Kim S, Choi S (2008) Diagnostic accuracy of ultrasound features in thyroid microcarcinomas. Endocr J 55:931–938

  32. Kim E, Park C, Chung W et al (2002) New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol 178:687–691

    Article  Google Scholar 

  33. Moon W, Jung S, Lee J et al (2008) Benign and malignant thyroid nodules: US differentiation--multicenter retrospective study. Radiology 247:762–770

    Article  Google Scholar 

  34. Moon W, Baek J, Jung S et al (2011) Ultrasonography and the ultrasound-based management of thyroid nodules: consensus statement and recommendations. Korean J Radiology 12:1–14

    Article  Google Scholar 

  35. Lee J, Lee S, Cha S, Cho B, Kang M, Lee O (2013) Fine-needle aspiration of thyroid nodules with macrocalcification. Thyroid 23:1106–1112

    Article  Google Scholar 

  36. Xu T, Gu J, Ye X et al (2017) Thyroid nodule sizes influence the diagnostic performance of TIRADS and ultrasound patterns of 2015 ATA guidelines: a multicenter retrospective study. Sci Rep 7:43183

  37. Trimboli P, Treglia G, Guidobaldi L et al (2014) Clinical characteristics as predictors of malignancy in patients with indeterminate thyroid cytology: a meta-analysis. Endocrine 46:52–59

    Article  CAS  Google Scholar 

  38. Ahn H, Na D, Baek J, Sung J, Kim J (2019) False negative rate of fine-needle aspiration in thyroid nodules: impact of nodule size and ultrasound pattern. Head Neck 41:967–973

Download references

Funding

The authors state that this work has not received any funding.

Author information

Author notes

  1. Ying Liu and Xiaoxian Li contributed equally to this work.

Authors and Affiliations

  1. Department of Ultrasound, Sun Yat-sen University Cancer Center, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, No. 651 Dongfeng Road East, Guangzhou, 510060, People’s Republic of China

    Ying Liu, Xiaoxian Li, Cuiju Yan, Longzhong Liu, Ying Liao & Jianhua Zhou

  2. Department of Ultrasound, Huadu District People’s Hospital, No. 48, Xinhua Road, Huadu District, Guangzhou, 510800, People’s Republic of China

    Hongyan Zeng

  3. Department of Ultrasound, Foshan First Municipal People’s Hospital (The Affiliated Foshan Hospital of Sun Yat-Sen University), No. 81 Lingn an North Road, Foshan, 528000, Guangdong Province, People’s Republic of China

    Weijun Huang

  4. Department of Ultrasound, Affiliated Tumor Hospital of Zhengzhou University, No. 127, Dongming Road, Jinshui District, Zhengzhou, 450008, People’s Republic of China

    Qian Li

  5. Department of Ultrasound, The Fifth People’s Hospital of Nanhai, No. 4, Dongyi street, Zhenxing Road, Lishui Town, Nanhai District, Foshan City, Guangdong Province, People’s Republic of China

    Nansheng Tao

Authors
  1. Ying Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoxian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cuiju Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Longzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ying Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongyan Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Weijun Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Qian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Nansheng Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jianhua Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianhua Zhou.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Jianhua Zhou.

Conflict of Interest

The authors of this manuscript declare no relationships with any companies, whose productsor services may be related to the subject matter of the article.

Statistics and Biometry

No complex statistical methods were necessary for this paper.

Informed Consent

Written informed consent was waived by the Institutional Review Board.

Ethical Approval

Institutional Review Board approval was obtained.

Methodology

• retrospective

• diagnostic study

• multicenter study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 34 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Li, X., Yan, C. et al. Comparison of diagnostic accuracy and utility of artificial intelligence–optimized ACR TI-RADS and original ACR TI-RADS: a multi-center validation study based on 2061 thyroid nodules. Eur Radiol 32, 7733–7742 (2022). https://doi.org/10.1007/s00330-022-08827-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-08827-y

Keywords

Search

Navigation