Skip to main content
SpringerLink
Log in

Combining radiomics with ultrasound-based risk stratification systems for thyroid nodules: an approach for improving performance

European Radiology Aims and scope Submit manuscript

Cite this article

Abstract

Objectives

To develop a radiomics score using ultrasound images to predict thyroid malignancy and to investigate its potential as a complementary tool to improve the performance of risk stratification systems.

Methods

We retrospectively included consecutive patients who underwent fine-needle aspiration (FNA) for thyroid nodules that were cytopathologically diagnosed as benign or malignant. Nodules were randomly assigned to a training and test set (8:2 ratio). A radiomics score was developed from the training set, and cutoff values based on the maximum Youden index (Rad_maxY) and for 5%, 10%, and 20% predicted malignancy risk (Rad_5%, Rad_10%, Rad_20%, respectively) were applied to the test set. The performances of the American College of Radiology (ACR) and the American Thyroid Association (ATA) guidelines were compared with the combined performances of the guidelines and radiomics score with interpretations from expert and nonexpert readers.

Results

A total of 1624 thyroid nodules from 1609 patients (mean age, 50.1 years [range, 18–90 years]) were included. The radiomics score yielded an AUC of 0.85 (95% CI: 0.83, 0.87) in the training set and 0.75 (95% CI: 0.69, 0.81) in the test set (Rad_maxY). When the radiomics score was combined with the ACR or ATA guidelines (Rad_5%), all readers showed increased specificity, accuracy, and PPV and decreased unnecessary FNA rates (all p < .05), with no difference in sensitivity (p > .05).

Conclusion

Radiomics help predict thyroid malignancy and improve specificity, accuracy, PPV, and unnecessary FNA rate while maintaining the sensitivity of the ACR and ATA guidelines for both expert and nonexpert readers.

Key Points

• The radiomics score yielded an AUC of 0.85 and 0.75 in the training and test set, respectively.

• For all readers, combining a 5% predicted malignancy risk cutoff for the radiomics score with the ACR and ATA guidelines significantly increased specificity, accuracy, and PPV and decreased unnecessary FNA rates, with no decrease in sensitivity.

• Radiomics can help predict malignancy in thyroid nodules in combination with risk stratification systems, by improving specificity, accuracy, and PPV and unnecessary FNA rates while maintaining sensitivity for both expert and nonexpert readers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Abbreviations

ACR:

American College of Radiology

ATA:

American Thyroid Association

AUC:

Area under the receiver operating characteristic curve

FNA:

Fine-needle aspiration

NPV:

Negative predictive value

PPV:

Positive predictive value

References

  1. Cronan JJ (2008) Thyroid nodules: is it time to turn off the US machines? Radiology 247:602–604

    Article  Google Scholar 

  2. Gnarini VL, Brigante G, Della Valle E et al (2013) Very high prevalence of ultrasound thyroid scan abnormalities in healthy volunteers in Modena, Italy. J Endocrinol Invest 36:722–728

    CAS  PubMed  Google Scholar 

  3. Guth S, Theune U, Aberle J, Galach A, Bamberger CM (2009) Very high prevalence of thyroid nodules detected by high frequency (13 MHz) ultrasound examination. Eur J Clin Invest 39:699–706

    Article  CAS  Google Scholar 

  4. Tan GH, Gharib H (1997) Thyroid incidentalomas: management approaches to nonpalpable nodules discovered incidentally on thyroid imaging. Ann Intern Med 126:226–231

    Article  CAS  Google Scholar 

  5. Vaccarella S, Dal Maso L, Laversanne M, Bray F, Plummer M, Franceschi S (2015) The impact of diagnostic changes on the rise in thyroid cancer incidence: a population-based study in selected high-resource countries. Thyroid 25:1127–1136

    Article  Google Scholar 

  6. Haugen BR, Alexander EK, Bible KC et al (2016) 2015 American Thyroid Association Management Guidelines for Adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on thyroid nodules and differentiated thyroid cancer. Thyroid 26:1–133

    Article  Google Scholar 

  7. Russ G, Bonnema SJ, Erdogan MF, Durante C, Ngu R, Leenhardt L (2017) European Thyroid Association Guidelines for ultrasound malignancy risk stratification of thyroid nodules in adults: The EU-TIRADS. Eur Thyroid J 6:225–237

    Article  Google Scholar 

  8. Shin JH, Baek JH, Chung J et al (2016) Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology Consensus Statement and Recommendations. Korean J Radiol 17:370–395

    Article  Google Scholar 

  9. Tessler FN, Middleton WD, Grant EG (2018) Thyroid imaging reporting and data system (TI-RADS): a user's guide. Radiology 287:29–36

    Article  Google Scholar 

  10. Tessler FN, Middleton WD, Grant EG et al (2017) ACR thyroid imaging, Reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee. J Am Coll Radiol 14:587–595

    Article  Google Scholar 

  11. Ha EJ, Na DG, Moon WJ, Lee YH, Choi N (2018) Diagnostic performance of ultrasound-based risk-stratification systems for thyroid nodules: comparison of the 2015 American Thyroid Association Guidelines with the 2016 Korean Thyroid Association/Korean Society of Thyroid Radiology and 2017 American College of Radiology Guidelines. Thyroid 28:1532–1537

    Article  Google Scholar 

  12. Middleton WD, Teefey SA, Reading CC et al (2018) Comparison of performance characteristics of American College of Radiology TI-RADS, Korean Society of Thyroid Radiology TIRADS, and American Thyroid Association Guidelines. AJR Am J Roentgenol 210:1148–1154

    Article  Google Scholar 

  13. Ruan JL, Yang HY, Liu RB et al (2019) Fine needle aspiration biopsy indications for thyroid nodules: compare a point-based risk stratification system with a pattern-based risk stratification system. Eur Radiol 29:4871–4878

    Article  Google Scholar 

  14. Yoon JH, Han K, Kim EK, Moon HJ, Kwak JY (2017) Diagnosis and management of small thyroid nodules: a comparative study with six guidelines for thyroid nodules. Radiology 283:560–569

    Article  Google Scholar 

  15. Yoon JH, Lee HS, Kim EK, Moon HJ, Park VY, Kwak JY (2020) Pattern-based vs. score-based guidelines using ultrasound features have different strengths in risk stratification of thyroid nodules. Eur Radiol. https://doi.org/10.1007/s00330-020-06722-y

  16. Ha EJ, Na DG, Baek JH, Sung JY, Kim JH, Kang SY (2018) US fine-needle aspiration biopsy for thyroid malignancy: diagnostic performance of seven society guidelines applied to 2000 thyroid nodules. Radiology 287:893–900

    Article  Google Scholar 

  17. Grani G, Lamartina L, Ascoli V et al (2019) Reducing the number of unnecessary thyroid biopsies while improving diagnostic accuracy: toward the "right" TIRADS. J Clin Endocrinol Metab 104:95–102

    Article  Google Scholar 

  18. Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577

    Article  Google Scholar 

  19. Liang J, Huang X, Hu H et al (2018) Predicting malignancy in thyroid nodules: radiomics score versus 2017 American College of Radiology Thyroid Imaging, Reporting and Data System. Thyroid 28:1024–1033

    Article  CAS  Google Scholar 

  20. Park VY, Han K, Seong YK et al (2019) Diagnosis of thyroid nodules: performance of a deep learning convolutional neural network model vs. radiologists. Sci Rep 9:17843

    Article  Google Scholar 

  21. Ha SM, Baek JH, Na DG et al (2019) Diagnostic performance of practice guidelines for thyroid nodules: thyroid nodule size versus biopsy rates. Radiology 291:92–99

    Article  Google Scholar 

  22. Yoon JH, Lee HS, Kim EK, Moon HJ, Kwak JY (2016) Malignancy risk stratification of thyroid nodules: comparison between the Thyroid Imaging Reporting and Data System and the 2014 American Thyroid Association Management Guidelines. Radiology 278:917–924

    Article  Google Scholar 

  23. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc B Methodol 58:267–288

    Google Scholar 

  24. Bhatia KS, Lam AC, Pang SW, Wang D, Ahuja AT (2016) Feasibility Study of texture analysis using ultrasound shear wave elastography to predict malignancy in thyroid nodules. Ultrasound Med Biol 42:1671–1680

    Article  Google Scholar 

  25. Chen SJ, Chang CY, Chang KY et al (2010) Classification of the thyroid nodules based on characteristic sonographic textural feature and correlated histopathology using hierarchical support vector machines. Ultrasound Med Biol 36:2018–2026

    Article  Google Scholar 

  26. Galimzianova A, Siebert SM, Kamaya A, Rubin DL, Desser TS (2020) Quantitative framework for risk stratification of thyroid nodules with ultrasound: a step toward automated triage of thyroid cancer. AJR Am J Roentgenol 214:885–892

    Article  Google Scholar 

  27. Nam SJ, Yoo J, Lee HS et al (2016) Quantitative evaluation for differentiating malignant and benign thyroid nodules using histogram analysis of grayscale sonograms. J Ultrasound Med 35:775–782

    Article  Google Scholar 

  28. Song G, Xue F, Zhang C (2015) A model using texture features to differentiate the nature of thyroid nodules on sonography. J Ultrasound Med 34:1753–1760

    Article  Google Scholar 

  29. Wildman-Tobriner B, Buda M, Hoang JK et al (2019) Using artificial intelligence to revise ACR TI-RADS risk stratification of thyroid nodules: diagnostic accuracy and utility. Radiology 292:112–119

    Article  Google Scholar 

  30. Buda M, Wildman-Tobriner B, Castor K, Hoang JK, Mazurowski MA (2020) Deep learning-based segmentation of nodules in thyroid ultrasound: improving performance by utilizing markers present in the images. Ultrasound Med Biol 46:415–421

    Article  Google Scholar 

Download references

Acknowledgments

This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2019R1A2C1002375). The funders had no role in study design, data collection, and analysis; decision to publish; or preparation of the manuscript.

Funding

This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2019R1A2C1002375). The funders had no role in study design, data collection, and analysis; decision to publish; or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University, College of Medicine, 50 Yonsei-ro, Seodaemun-gu, 03722, Seoul, Korea

    Vivian Y. Park, Jiyoung Yoon, Jinwoo Son, Hee Jung Moon, Jung Hyun Yoon, Ga Ram Kim & Jin Young Kwak

  2. Department of Computational Science and Engineering, Yonsei University, Seoul, Korea

    Eunjung Lee

  3. Biostatistics Collaboration Unit, Yonsei University, College of Medicine, Seoul, Korea

    Hye Sun Lee

  4. Department of Radiology, Kyungpook National University Chilgok Hospital, School of Medicine, Kyungpook National University, Daegu, Korea

    Hye Jung Kim

  5. Department of Biostatistics, Yonsei University, College of Nursing, Seoul, Korea

    Kijun Song

Authors
  1. Vivian Y. Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eunjung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hye Sun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hye Jung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiyoung Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jinwoo Son
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kijun Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hee Jung Moon
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jung Hyun Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ga Ram Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jin Young Kwak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Young Kwak.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Jin Young Kwak, M.D., PhD.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors (Hye Sun Lee, PhD ) has significant statistical expertise.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Among our study population, 507 thyroid nodules in 507 patients were included in a previous study which developed and evaluated the performance of a deep learning–based US computer-aided diagnosis system (Reference: Park VY, Han K, Seong YK et al (2019) Diagnosis of Thyroid Nodules: Performance of a deep learning convolutional neural network model vs. radiologists. Sci Rep 9:17843.).

Methodology

• retrospective

• diagnostic or prognostic study

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOCX 743 kb)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, V.Y., Lee, E., Lee, H.S. et al. Combining radiomics with ultrasound-based risk stratification systems for thyroid nodules: an approach for improving performance. Eur Radiol 31, 2405–2413 (2021). https://doi.org/10.1007/s00330-020-07365-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-020-07365-9

Keywords

search

Navigation