pp 1–10 | Cite as

Ultrasound characterization for thyroid nodules with indeterminate cytology: inter-observer agreement and impact of combining pattern-based and scoring-based classifications in risk stratification

  • Cesar A. LamEmail author
  • Melissa J. McGettigan
  • Zachary J. Thompson
  • Laila Khazai
  • Christine H. Chung
  • Barbara A. Centeno
  • Bryan McIver
  • Pablo Valderrabano
Original Article



The American Thyroid Association (ATA) sonographic patterns stratify the risk of malignancy of cytologically indeterminate thyroid nodules (ITNs). This study aimed to (1) assess inter-observer agreement for sonographic features and patterns; (2) identify potential sources of disagreement; and (3) evaluate whether the number of suspicious features risk-stratifies non-ATA and high-suspicion patterns.


Three observers independently reviewed the ultrasound images of 463 ITNs with histological follow-up consecutively evaluated between October 2008 and June 2015 at an academic cancer center. Each observer evaluated individual sonographic features. ATA sonographic patterns were derived from the interpretation of sonographic features. Nodules not fitting into any of the proposed patterns were clustered into a non-ATA pattern.


The inter-observer agreement for ATA sonographic patterns and echogenicity was fair, moderate for margins, good for composition and echogenic foci, and very good for extrathyroidal extension and lymph node metastasis. The interpretation of each sonographic feature was significantly different between observers, and there was complete disagreement in at least one of the features in 104 (22%) nodules. A total of 169 nodules (37%) were classified into the non-ATA pattern. The number of suspicious features allowed risk stratifying nodules with non-ATA and high-suspicion sonographic patterns. Most Non-invasive Follicular Thyroid Neoplasms with Papillary-like Nuclear Features had 0–1 suspicious features and none had >2.


Echogenicity interpretation was the greatest source of disagreement. The number of suspicious features risk-stratifies ITNs with non-ATA or high-suspicion patterns. Future studies attempting to objectivize the interpretation of echogenicity and heterogeneity are needed.


Thyroid ultrasound Thyroid cytology Thyroid nodules Thyroid cancer Non-invasive follicular thyroid neoplasm with papillary-like nuclear features (NIFTP) 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

The study was carried with a waiver of informed consent from our Institutional Review Board.

Supplementary material

12020_2019_2000_MOESM1_ESM.docx (14 kb)
Supplementary Table


  1. 1.
    H.S. Ahn, H.J. Kim, H.G. Welch, Korea’s thyroid-cancer “epidemic”-screening and overdiagnosis. N. Engl. J. Med. 371(19), 1765–1767 (2014). Google Scholar
  2. 2.
    S. Nagar, B. Aschebrook-Kilfoy, E.L. Kaplan, P. Angelos, R.H. Grogan, Age of diagnosing physician impacts the incidence of thyroid cancer in a population. Cancer Causes Control 25(12), 1627–1634 (2014). Google Scholar
  3. 3.
    R. Udelsman, Y. Zhang, The epidemic of thyroid cancer in the United States: the role of endocrinologists and ultrasounds. Thyroid 24(3), 472–479 (2014). Google Scholar
  4. 4.
    S. Vaccarella, S. Franceschi, F. Bray, C.P. Wild, M. Plummer, L. Dal Maso, Worldwide thyroid-cancer epidemic? The increasing impact of overdiagnosis. N. Engl. J. Med. 375(7), 614–617 (2016). Google Scholar
  5. 5.
    J.P. Zevallos, C.M. Hartman, J.R. Kramer, E.M. Sturgis, E.Y. Chiao, Increased thyroid cancer incidence corresponds to increased use of thyroid ultrasound and fine-needle aspiration: a study of the Veterans Affairs health care system. Cancer 121(5), 741–746 (2015). Google Scholar
  6. 6.
    J.A. Sosa, J.W. Hanna, K.A. Robinson, R.B. Lanman, Increases in thyroid nodule fine-needle aspirations, operations, and diagnoses of thyroid cancer in the United States. Surgery 154(6), 1420–1426 (2013). discussion 1426–1427Google Scholar
  7. 7.
    B.R. Haugen, E.K. Alexander, K.C. Bible, G.M. Doherty, S.J. Mandel, Y.E. Nikiforov, F. Pacini, G.W. Randolph, A.M. Sawka, M. Schlumberger, K.G. Schuff, S.I. Sherman, J.A. Sosa, D.L. Steward, R.M. Tuttle, L. Wartofsky, 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 26(1), 1–133 (2016). Google Scholar
  8. 8.
    P. Campanella, F. Ianni, C.A. Rota, S.M. Corsello, A. Pontecorvi, Quantification of cancer risk of each clinical and ultrasonographic suspicious feature of thyroid nodules: a systematic review and meta-analysis. Eur. J. Endocrinol. / Eur. Fed. Endocr. Soc. 170(5), R203–R211 (2014). Google Scholar
  9. 9.
    L.R. Remonti, C.K. Kramer, C.B. Leitao, L.C. Pinto, J.L. Gross, Thyroid ultrasound features and risk of carcinoma: a systematic review and meta-analysis of observational studies. Thyroid 25(5), 538–550 (2015). Google Scholar
  10. 10.
    J.P. Brito, M.R. Gionfriddo, A. Al Nofal, K.R. Boehmer, A.L. Leppin, C. Reading, M. Callstrom, T.A. Elraiyah, L.J. Prokop, M.N. Stan, M.H. Murad, J.C. Morris, V.M. Montori, The accuracy of thyroid nodule ultrasound to predict thyroid cancer: systematic review and meta-analysis. J. Clin. Endocrinol. Metab. 99(4), 1253–1263 (2014). Google Scholar
  11. 11.
    W.J. Moon, S.L. Jung, J.H. Lee, D.G. Na, J.H. Baek, Y.H. Lee, J. Kim, H.S. Kim, J.S. Byun, D.H. Lee, Benign and malignant thyroid nodules: US differentiation-multicenter retrospective study. Radiology 247(3), 762–770 (2008). Google Scholar
  12. 12.
    C.S. Park, S.H. Kim, S.L. Jung, B.J. Kang, J.Y. Kim, J.J. Choi, M.S. Sung, H.W. Yim, S.H. Jeong, Observer variability in the sonographic evaluation of thyroid nodules. J. Clin. Ultrasound 38(6), 287–293 (2010). Google Scholar
  13. 13.
    P. Valderrabano, D.L. Klippenstein, J.B. Tourtelot, Z. Ma, Z.J. Thompson, H.S. Lilienfeld, B. McIver, New American Thyroid Association Sonographic Patterns for Thyroid Nodules Perform Well in Medullary Thyroid Carcinoma: Institutional Experience, Systematic Review, and Meta-Analysis. Thyroid 26(8), 1093–1100 (2016). Google Scholar
  14. 14.
    P. Valderrabano, M.J. McGettigan, C.A. Lam, L. Khazai, Z.J. Thompson, C.H. Chung, B.A. Centeno, B. McIver, Thyroid nodules with indeterminate cytology: utility of the American Thyroid Association Sonographic Patterns for Cancer Risk Stratification. Thyroid 28(8), 1004–1012 (2018). Google Scholar
  15. 15.
    E.S. Cibas, S.Z. Ali, The Bethesda system for reporting thyroid cytopathology. Thyroid 19(11), 1159–1165 (2009). Google Scholar
  16. 16.
    Randolph, J. Free-marginal multirater kappa: an alternative to Fleiss’ fixed-marginal multirater kappa. Joensuu University Learning and Instruction Symposium, Joensuu, Finland, 2005.Google Scholar
  17. 17.
    M.J. Warrens, Inequalities between multi-rater kappas. Adv. Data Anal. Classif. 4(4), 271–286 (2010). Google Scholar
  18. 18.
    S.H. Choi, E.K. Kim, J.Y. Kwak, M.J. Kim, E.J. Son, Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. Thyroid 20(2), 167–172 (2010). Google Scholar
  19. 19.
    S.H. Park, S.J. Kim, E.K. Kim, M.J. Kim, E.J. Son, J.Y. Kwak, Interobserver agreement in assessing the sonographic and elastographic features of malignant thyroid nodules. AJR Am. J. Roentgenol. 193(5), W416–W423 (2009). Google Scholar
  20. 20.
    G. Grani, L. Lamartina, V. Cantisani, M. Maranghi, P. Lucia, C. Durante, Interobserver agreement of various thyroid imaging reporting and data systems. Endocr. Connect 7(1), 1–7 (2018). Google Scholar
  21. 21.
    W. Phuttharak, A. Boonrod, V. Klungboonkrong, T. Witsawapaisan, Interrater Reliability of Various Thyroid Imaging Reporting and Data System (TIRADS) Classifications for Differentiating Benign from Malignant Thyroid Nodules. Asian Pac. J. Cancer Prev. 20(4), 1283–1288 (2019). Google Scholar
  22. 22.
    G. Russ, S.J. Bonnema, M.F. Erdogan, C. Durante, R. Ngu, L. Leenhardt, European Thyroid Association Guidelines for Ultrasound Malignancy Risk Stratification of Thyroid Nodules in Adults: The EU-TIRADS. Eur. Thyroid J. 6(5), 225–237 (2017). Google Scholar
  23. 23.
    P. Trimboli, M. Deandrea, A. Mormile, L. Ceriani, F. Garino, P.P. Limone, L. Giovanella, American Thyroid Association ultrasound system for the initial assessment of thyroid nodules: use in stratifying the risk of malignancy of indeterminate lesions. Head. Neck 40(4), 722–727 (2018). Google Scholar
  24. 24.
    T.G. Rocha, P.W. Rosario, A.L. Silva, M.B. Nunes, T.H. Silva, P.H.L. de Oliveira, M.R. Calsolari, Ultrasonography Classification of the American Thyroid Association for Predicting Malignancy in Thyroid Nodules >1cm with Indeterminate Cytology: A Prospective Study. Horm. Metab. Res 50(8), 597–601 (2018). Google Scholar
  25. 25.
    G. Grani, M. D’Alessandri, G. Carbotta, A. Nesca, M. Del Sordo, S. Alessandrini, C. Coccaro, R. Rendina, M. Bianchini, N. Prinzi, A. Fumarola, Grey-scale analysis improves the ultrasonographic evaluation of thyroid nodules. Med. (Baltim.) 94(27), e1129 (2015). Google Scholar
  26. 26.
    L. Gao, X. Xi, J. Wang, X. Yang, Y. Wang, S. Zhu, X. Lai, X. Zhang, R. Zhao, B. Zhang, Ultrasound risk evaluation of thyroid nodules that are “unspecified” in the 2015 American Thyroid Association management guidelines: a retrospective study. Med. (Baltim.) 97(52), e13914 (2018). Google Scholar
  27. 27.
    J.E. Lim-Dunham, I. Erdem Toslak, K. Alsabban, A. Aziz, B. Martin, G. Okur, K.C. Longo, Ultrasound risk stratification for malignancy using the 2015 American Thyroid Association Management Guidelines for Children with Thyroid Nodules and Differentiated Thyroid Cancer. Pedia. Radio. 47(4), 429–436 (2017). Google Scholar
  28. 28.
    G. Grani, L. Lamartina, V. Ascoli, D. Bosco, F. Nardi, F. D’Ambrosio, A. Rubini, L. Giacomelli, M. Biffoni, S. Filetti, C. Durante, V. Cantisani, Ultrasonography scoring systems can rule out malignancy in cytologically indeterminate thyroid nodules. Endocrine 57(2), 256–261 (2017). Google Scholar
  29. 29.
    G. Russ, S. Leboulleux, L. Leenhardt, L. Hegedüs, Thyroid incidentalomas: epidemiology, risk stratification with ultrasound and workup. Eur. Thyroid J. 3(3), 154–163 (2014). Google Scholar
  30. 30.
    M. Sollini, L. Cozzi, A. Chiti, M. Kirienko, Texture analysis and machine learning to characterize suspected thyroid nodules and differentiated thyroid cancer: Where do we stand? Eur. J. Radio. 99, 1–8 (2018). Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Cesar A. Lam
    • 1
    Email author
  • Melissa J. McGettigan
    • 1
  • Zachary J. Thompson
    • 2
  • Laila Khazai
    • 3
  • Christine H. Chung
    • 4
  • Barbara A. Centeno
    • 3
  • Bryan McIver
    • 4
  • Pablo Valderrabano
    • 4
    • 5
  1. 1.Department of Diagnostic ImagingH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  2. 2.Department of Biostatistics and BioinformaticsH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  3. 3.Department of Anatomic PathologyH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  4. 4.Department of Head and Neck-Endocrine OncologyH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  5. 5.Department of Endocrinology and NutritionHospital Universitario Ramón y Cajal, IRYCISMadridSpain

Personalised recommendations