Abstract
Purpose
To investigate the interobserver agreement (IA) and the impact of consensus reading using four risk stratification systems for thyroid nodules (TN).
Methods
Four experienced specialists independently rated US images of 80 TN according to the Kwak-TIRADS, EU-TIRADS, ACR TI-RADS, and ATA Guidelines. The cases were randomly extracted from a prospectively acquired database (n > 1500 TN). The observers were blinded to clinical data. This study was divided into two sessions (S1 and S2) with 40 image sets each. After every session, a consensus reading was carried out (C1, C2). Subsequently, the effect of C1 was tested in S2 with 40 new cases followed by C2. Fleiss’ kappa (κ) was calculated for S1 and S2 to estimate the IA and learning curves. The results of C1 and C2 were used as reference for diagnostic accuracy calculations.
Results
IA significantly increased (p < 0.01) after C1 with κ values of 0.375 (0.615), 0.411 (0.596), 0.321 (0.569), and 0.410 (0.583) for the Kwak-TIRADS, EU-TIRADS, ACR TI-RADS, and ATA Guidelines in S1 (S2), respectively. ROC analysis (C1 + C2) revealed similar areas under the curve (AUC) for the Kwak-TIRADS, EU-TIRADS, ACR TI-RADS, and ATA Guidelines (0.635, 0.675, 0.694, and 0.654, respectively, n.s.). AUC did not increase from C1 (0.677 ± 0.010) to C2 (0.632 ± 0.052, n.s.). ATA Guidelines were not applicable in five cases.
Conclusions
IA and diagnostic accuracy were very similar for the four investigated risk stratification systems. Consensus reading sessions significantly improved the IA but did not affect the diagnostic accuracy.
Similar content being viewed by others
References
C. Reiners, K. Wegscheider, H. Schicha, P. Theissen, R. Vaupel, R. Wrbitzky, P.M. Schumm-Draeger, Prevalence of thyroid disorders in the working population of Germany: ultrasonography screening in 96,278 unselected employees. Thyroid 14(11), 926–932 (2004). https://doi.org/10.1089/thy.2004.14.926
G. Russ, S. Leboulleux, L. Leenhardt, L. Hegedus, Thyroid incidentalomas: epidemiology, risk stratification with ultrasound and workup. Eur. Thyroid J. 3(3), 154–163 (2014). https://doi.org/10.1159/000365289
S. Guth, U. Theune, J. Aberle, A. Galach, C.M. Bamberger, Very high prevalence of thyroid nodules detected by high frequency (13 MHz) ultrasound examination. Eur. J. Clin. Invest. 39(8), 699–706 (2009). https://doi.org/10.1111/j.1365-2362.2009.02162.x
F. Bray, J. Ferlay, I. Soerjomataram, R.L. Siegel, L.A. Torre, A. Jemal, Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 68(6), 394–424 (2018). https://doi.org/10.3322/caac.21492
C.M. Oh, K.W. Jung, Y.J. Won, A. Shin, H.J. Kong, J.S. Lee, Age-period-cohort analysis of thyroid cancer incidence in Korea. Cancer Res Treat. 47(3), 362–369 (2015). https://doi.org/10.4143/crt.2014.110
L. Liang, X.C. Zheng, M.J. Hu, Q. Zhang, S.Y. Wang, F. Huang, Association of benign thyroid diseases with thyroid cancer risk: a meta-analysis of prospective observational studies. J. Endocrinol. Invest. 42(6), 673–685 (2019). https://doi.org/10.1007/s40618-018-0968-z
J.Y. Kwak, K.H. Han, J.H. Yoon, H.J. Moon, E.J. Son, S.H. Park, H.K. Jung, J.S. Choi, B.M. Kim, E.K. Kim, Thyroid imaging reporting and data system for US features of nodules: a step in establishing better stratification of cancer risk. Radiology 260(3), 892–899 (2011). https://doi.org/10.1148/radiol.11110206
E.J. Ha, J.H. Baek, D.G. Na, Risk stratification of thyroid nodules on ultrasonography: current status and perspectives. Thyroid 27(12), 1463–1468 (2017). https://doi.org/10.1089/thy.2016.0654
S.M. Ha, H.S. Ahn, J.H. Baek, H.Y. Ahn, Y.J. Chung, B.Y. Cho, S.B. Park, Validation of three scoring risk-stratification models for thyroid nodules. Thyroid 27(12), 1550–1557 (2017). https://doi.org/10.1089/thy.2017.0363
S. Schenke, T. Rink, M. Zimny, TIRADS for sonographic assessment of hypofunctioning and indifferent thyroid nodules. Nuklearmedizin 54(3), 144–150 (2015). https://doi.org/10.3413/Nukmed-0712-14-12
J.Y. Park, H.J. Lee, H.W. Jang, H.K. Kim, J.H. Yi, W. Lee, S.H. Kim, A proposal for a thyroid imaging reporting and data system for ultrasound features of thyroid carcinoma. Thyroid 19(11), 1257–1264 (2009). https://doi.org/10.1089/thy.2008.0021
B.R. Haugen, E.K. Alexander, K.C. Bible, G.M. Doherty, S.J. Mandel, Y.E. Nikiforov, F. Pacini, G.W. Randolph, A.M. Sawka, M. Schlumberger, K.G. Schuff, S.I. Sherman, J.A. Sosa, D.L. Steward, R.M. Tuttle, L. Wartofsky, 2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on thyroid nodules and differentiated thyroid cancer. Thyroid 26(1), 1–133 (2016). https://doi.org/10.1089/thy.2015.0020
S. Schenke, P. Seifert, M. Zimny, T. Winkens, I. Binse, R. Goerges, Risk stratification of thyroid nodules using Thyroid Imaging Reporting And Data System (TIRADS): the omission of thyroid scintigraphy increases the rate of falsely suspected lesions. J. Nucl. Med. 60(3), 342–347 (2019). https://doi.org/10.2967/jnumed.118.211912
E. Horvath, S. Majlis, R. Rossi, C. Franco, J.P. Niedmann, A. Castro, M. Dominguez, An ultrasonogram reporting system for thyroid nodules stratifying cancer risk for clinical management. J. Clin. Endocrinol. Metab. 94(5), 1748–1751 (2009). https://doi.org/10.1210/jc.2008-1724
P. Perros, K. Boelaert, S. Colley, C. Evans, R.M. Evans, G. Gerrard Ba, J. Gilbert, B. Harrison, S.J. Johnson, T.E. Giles, L. Moss, V. Lewington, K. Newbold, J. Taylor, R.V. Thakker, J. Watkinson, G.R. Williams, A. British Thyroid, Guidelines for the management of thyroid cancer. Clin. Endocrinol. (Oxf.) 81(Suppl 1), 1–122 (2014). https://doi.org/10.1111/cen.12515
E.J. Ha, W.J. Moon, D.G. Na, Y.H. Lee, N. Choi, S.J. Kim, J.K. Kim, A multicenter prospective validation study for the Korean thyroid imaging reporting and data system in patients with thyroid nodules. Korean J. Radio. 17(5), 811–821 (2016). https://doi.org/10.3348/kjr.2016.17.5.811
G. Russ, S.J. Bonnema, M.F. Erdogan, C. Durante, R. Ngu, L. Leenhardt, European thyroid association guidelines for ultrasound malignancy risk stratification of thyroid nodules in adults: the EU-TIRADS. Eur. Thyroid J. 6(5), 225–237 (2017). https://doi.org/10.1159/000478927
F.N. Tessler, W.D. Middleton, E.G. Grant, J.K. Hoang, L.L. Berland, S.A. Teefey, J.J. Cronan, M.D. Beland, T.S. Desser, M.C. Frates, L.W. Hammers, U.M. Hamper, J.E. Langer, C.C. Reading, L.M. Scoutt, A.T. Stavros, ACR thyroid imaging, reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee. J. Am. Coll. Radio. 14(5), 587–595 (2017). https://doi.org/10.1016/j.jacr.2017.01.046
G. Grani, L. Lamartina, V. Cantisani, M. Maranghi, P. Lucia, C. Durante, Interobserver agreement of various thyroid imaging reporting and data systems. Endocr. Connect 7(1), 1–7 (2018). https://doi.org/10.1530/EC-17-0336
C.L. Chng, H.C. Tan, C.W. Too, W.Y. Lim, P.P.S. Chiam, L. Zhu, N.V. Nadkarni, A.Y.Y. Lim, Diagnostic performance of ATA, BTA and TIRADS sonographic patterns in the predication of malignancy in histologically proven thyroid nodules. Singap. Med J. 59(11), 578–583 (2018). https://doi.org/10.11622/smedj.2018062
S.H. Choi, E.K. Kim, J.Y. Kwak, M.J. Kim, E.J. Son, Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. Thyroid 20(2), 167–172 (2010). https://doi.org/10.1089/thy.2008.0354
P. Valderrabano, B. McIver, Evaluation and management of indeterminate thyroid nodules: the revolution of risk stratification beyond cytological diagnosis. Cancer Control 24(5), 1–14 (2017). https://doi.org/10.1177/1073274817729231
J.R. Landis, G.G. Koch, The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)
R. Wienhold, M. Scholz, J.R. Adler, G.N. C, R. Paschke, The management of thyroid nodules: a retrospective analysis of health insurance data. Dtsch. Arztebl. Int. 110(49), 827–834 (2013). https://doi.org/10.3238/arztebl.2013.0827
P. Seifert, M. Freesmeyer, Preoperative diagnostics in differentiated thyroid carcinoma. Nuklearmedizin 56(6), 201–210 (2017). https://doi.org/10.3413/Nukmed-0924-17-08
H. Gharib, E. Papini, J.R. Garber, D.S. Duick, R.M. Harrell, L. Hegedus, R. Paschke, R. Valcavi, P. Vitti, A.A.A.T.Fo.T. Nodules, American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules-2016 update. Endocr. Pract. 22(5), 622–639 (2016). https://doi.org/10.4158/EP161208.GL
H. Dralle, T.J. Musholt, J. Schabram, T. Steinmuller, A. Frilling, D. Simon, P.E. Goretzki, B. Niederle, C. Scheuba, T. Clerici, M. Hermann, J. Kussmann, K. Lorenz, C. Nies, P. Schabram, A. Trupka, A. Zielke, W. Karges, M. Luster, K.W. Schmid, D. Vordermark, H.J. Schmoll, R. Muhlenberg, O. Schober, H. Rimmele, A. Machens, German Societies of General and Visceral Surgery; Endocrinology; Nuclear Medicine; Pathology; Radiooncology; Oncological Hematology; and the German Thyroid Cancer Patient Support Organization Ohne Schilddrüse leben e.V., German Association of Endocrine Surgeons practice guideline for the surgical management of malignant thyroid tumors. Langenbecks Arch. Surg. 398(3), 347–375 (2013). https://doi.org/10.1007/s00423-013-1057-6
F. Pacini, M. Schlumberger, H. Dralle, R. Elisei, J.W. Smit, W. Wiersinga, European Thyroid Cancer Taskforce, European consensus for the management of patients with differentiated thyroid carcinoma of the follicular epithelium. Eur. J. Endocrinol. 154(6), 787–803 (2006). https://doi.org/10.1530/eje.1.02158
L. Leenhardt, M.F. Erdogan, L. Hegedus, S.J. Mandel, R. Paschke, T. Rago, G. Russ, 2013 European thyroid association guidelines for cervical ultrasound scan and ultrasound-guided techniques in the postoperative management of patients with thyroid cancer. Eur. Thyroid J. 2(3), 147–159 (2013). https://doi.org/10.1159/000354537
J. Koh, S.Y. Kim, H.S. Lee, E.K. Kim, J.Y. Kwak, H.J. Moon, J.H. Yoon, Diagnostic performances and interobserver agreement according to observer experience: a comparison study using three guidelines for management of thyroid nodules. Acta Radio. 59(8), 917–923 (2018). https://doi.org/10.1177/0284185117744001
J. Koh, H.J. Moon, J.S. Park, S.J. Kim, H.Y. Kim, E.K. Kim, J.Y. Kwak, Variability in interpretation of ultrasound elastography and gray-scale ultrasound in assessing thyroid nodules. Ultrasound Med. Biol. 42(1), 51–59 (2016). https://doi.org/10.1016/j.ultrasmedbio.2015.08.005
S.H. Park, S.J. Kim, E.K. Kim, M.J. Kim, E.J. Son, J.Y. Kwak, Interobserver agreement in assessing the sonographic and elastographic features of malignant thyroid nodules. AJR Am. J. Roentgenol. 193(5), W416–W423 (2009). https://doi.org/10.2214/AJR.09.2541
S.J. Park, S.H. Park, Y.J. Choi, D.W. Kim, E.J. Son, H.S. Lee, J.H. Yoon, E.K. Kim, H.J. Moon, J.Y. Kwak, Interobserver variability and diagnostic performance in US assessment of thyroid nodule according to size. Ultraschall Med. 33(7), E186–E190 (2012). https://doi.org/10.1055/s-0032-1325404
S.P. Cheng, J.J. Lee, J.L. Lin, S.M. Chuang, M.N. Chien, C.L. Liu, Characterization of thyroid nodules using the proposed thyroid imaging reporting and data system (TI-RADS). Head. Neck 35(4), 541–547 (2013). https://doi.org/10.1002/hed.22985
C.S. Park, S.H. Kim, S.L. Jung, B.J. Kang, J.Y. Kim, J.J. Choi, M.S. Sung, H.W. Yim, S.H. Jeong, Observer variability in the sonographic evaluation of thyroid nodules. J. Clin. Ultrasound 38(6), 287–293 (2010). https://doi.org/10.1002/jcu.20689
S.H. Kim, C.S. Park, S.L. Jung, B.J. Kang, J.Y. Kim, J.J. Choi, Y.I. Kim, J.K. Oh, J.S. Oh, H. Kim, S.H. Jeong, H.W. Yim, Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules. Korean J. Radio. 11(2), 149–155 (2010). https://doi.org/10.3348/kjr.2010.11.2.149
H.G. Kim, J.Y. Kwak, E.K. Kim, S.H. Choi, H.J. Moon, Man to man training: can it help improve the diagnostic performances and interobserver variabilities of thyroid ultrasonography in residents? Eur. J. Radio. 81(3), e352–e356 (2012). https://doi.org/10.1016/j.ejrad.2011.11.011
C. Anuradha, K. Abhishek, B. Pushpa, A. Deepak, Positive predictive value and inter-observer agreement of TI-RADS for ulrasound features of thyroid nodules. Paper presented at the Abstract presented at the European Congress of Radiology (ECR), Vienna, Austria, 6–10 March. Abstract no. C-0594 (2014).
A. Chandramohan, A. Khurana, B.T. Pushpa, M.T. Manipadam, D. Naik, N. Thomas, D. Abraham, M.J. Paul, Is TIRADS a practical and accurate system for use in daily clinical practice? Indian J. Radio. Imaging 26(1), 145–152 (2016). https://doi.org/10.4103/0971-3026.178367
M.N. Srinivas, V.N. Amogh, M.S. Gautam, I.S. Prathyusha, N.R. Vikram, M.K. Retnam, B.V. Balakrishna, N. Kudva, A prospective study to evaluate the reliability of thyroid imaging reporting and data system in differentiation between benign and malignant thyroid lesions. J. Clin. Imaging Sci. 6, 5 (2016). https://doi.org/10.4103/2156-7514.177551
J.M. Bae, S.Y. Hahn, J.H. Shin, E.Y. Ko, Inter-exam agreement and diagnostic performance of the Korean Thyroid Imaging Reporting and Data System for thyroid nodule assessment: real-time versus static ultrasonography. Eur. J. Radiol. 98, 14–19 (2018). https://doi.org/10.1016/j.ejrad.2017.10.027
J.M. Bae, S.Y. Hahn, J.H. Shin, E.Y. Ko, Erratum to “Inter-exam agreement and diagnostic performance of the Korean Thyroid Imaging Reporting and Data System for thyroid nodule assessment: real-time versus static ultrasonography” [Eur. J. Radiol. 98 (2018) 14–19]. Eur. J. Radiol. 101, 193 (2018).
J.K. Hoang, W.D. Middleton, A.E. Farjat, S.A. Teefey, N. Abinanti, F.J. Boschini, A.J. Bronner, N. Dahiya, B.S. Hertzberg, J.R. Newman, D. Scanga, R.C. Vogler, F.N. Tessler, Interobserver variability of sonographic features used in the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am. J. Roentgenol. 211(1), 162–167 (2018). https://doi.org/10.2214/AJR.17.19192
A. Persichetti, E. Di Stasio, R. Guglielmi, G. Bizzarri, S. Taccogna, I. Misischi, F. Graziano, L. Petrucci, A. Bianchini, E. Papini, Predictive value of malignancy of thyroid nodule ultrasound classification systems: a prospective study. J. Clin. Endocrinol. Metab. 103(4), 1359–1368 (2018). https://doi.org/10.1210/jc.2017-01708
G. Grani, M. D’Alessandri, G. Carbotta, A. Nesca, M. Del Sordo, S. Alessandrini, C. Coccaro, R. Rendina, M. Bianchini, N. Prinzi, A. Fumarola, Grey-scale analysis improves the ultrasonographic evaluation of thyroid nodules. Med. Baltim. 94(27), e1129 (2015). https://doi.org/10.1097/MD.0000000000001129
L. Gao, R. Liu, Y. Jiang, W. Song, Y. Wang, J. Liu, J. Wang, D. Wu, S. Li, A. Hao, B. Zhang, Computer-aided system for diagnosing thyroid nodules on ultrasound: A comparison with radiologist-based clinical assessments. Head. Neck 40(4), 778–783 (2018). https://doi.org/10.1002/hed.25049
S. Gitto, G. Grassi, C. De Angelis, C.G. Monaco, S. Sdao, F. Sardanelli, L.M. Sconfienza, G. Mauri, A computer-aided diagnosis system for the assessment and characterization of low-to-high suspicion thyroid nodules on ultrasound. Radio. Med 124(2), 118–125 (2019). https://doi.org/10.1007/s11547-018-0942-z
E.Y. Jeong, H.L. Kim, E.J. Ha, S.Y. Park, Y.J. Cho, M. Han, Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur. Radio. 29(4), 1978–1985 (2019). https://doi.org/10.1007/s00330-018-5772-9
Y.J. Yoo, E.J. Ha, Y.J. Cho, H.L. Kim, M. Han, S.Y. Kang, Computer-aided diagnosis of thyroid nodules via ultrasonography: initial clinical experience. Korean J. Radio. 19(4), 665–672 (2018). https://doi.org/10.3348/kjr.2018.19.4.665
K. Kobayashi, T. Fujimoto, H. Ota, M. Hirokawa, T. Yabuta, H. Masuoka, M. Fukushima, T. Higashiyama, M. Kihara, Y. Ito, A. Miya, A. Miyauchi, Calcifications in thyroid tumors on ultrasonography: calcification types and relationship with histopathological type. Ultrasound Int. Open 4(2), E45–E51 (2018). https://doi.org/10.1055/a-0591-6070
L. Gao, X. Xi, J. Wang, X. Yang, Y. Wang, S. Zhu, X. Lai, X. Zhang, R. Zhao, B. Zhang, Ultrasound risk evaluation of thyroid nodules that are “unspecified” in the 2015 American Thyroid Association management guidelines: a retrospective study. Med. Baltim. 97(52), e13914 (2018). https://doi.org/10.1097/MD.0000000000013914
P. Seifert, F. Guhne, M. Freesmeyer, Hyperfunctioning papillary microcarcinoma diagnosed by 124I PET/ultrasound fusion imaging. Clin. Nucl. Med. (2019). https://doi.org/10.1097/RLU.0000000000002503
M.J. Lima, V. Soares, P. Koch, A. Silva, A. Taveira-Gomes, Autonomously hyperfunctioning cystic nodule harbouring thyroid carcinoma - case report and literature review. Int J. Surg. Case Rep. 42, 287–289 (2018). https://doi.org/10.1016/j.ijscr.2018.01.002
G. Yun, Y.K. Kim, S.I. Choi, J.H. Kim, Medullary thyroid carcinoma: application of Thyroid Imaging Reporting and Data System (TI-RADS) Classification. Endocrine 61(2), 285–292 (2018). https://doi.org/10.1007/s12020-018-1594-4
M.J. Liu, Z.F. Liu, Y.Y. Hou, Y.M. Men, Y.X. Zhang, L.Y. Gao, H. Liu, Ultrasonographic characteristics of medullary thyroid carcinoma: a comparison with papillary thyroid carcinoma. Oncotarget 8(16), 27520–27528 (2017). https://doi.org/10.18632/oncotarget.15897
D.S. Kim, J.H. Kim, D.G. Na, S.H. Park, E. Kim, K.H. Chang, C.H. Sohn, Y.H. Choi, Sonographic features of follicular variant papillary thyroid carcinomas in comparison with conventional papillary thyroid carcinomas. J. Ultrasound Med. 28(12), 1685–1692 (2009)
E. Chaigneau, G. Russ, B. Royer, C. Bigorgne, M. Bienvenu-Perrard, A. Rouxel, L. Leenhardt, L. Belin, C. Buffet, TIRADS score is of limited clinical value for risk stratification of indeterminate cytological results. Eur. J. Endocrinol. 179(1), 13–20 (2018). https://doi.org/10.1530/EJE-18-0078
H. Baser, B. Cakir, O. Topaloglu, A. Alkan, S.B. Polat, H.T. Dogan, M.O. Yazicioglu, C. Aydin, R. Ersoy, Diagnostic accuracy of Thyroid Imaging Reporting and Data System in the prediction of malignancy in nodules with atypia and follicular lesion of undetermined significance cytologies. Clin. Endocrinol. Oxf. 86(4), 584–590 (2017). https://doi.org/10.1111/cen.13274
Acknowledgements
The authors thank PD Dr med. Martin Freesmeyer of the Department of Nuclear Medicine at Jena University Hospital for providing access to patient data and images.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Seifert, P., Görges, R., Zimny, M. et al. Interobserver agreement and efficacy of consensus reading in Kwak-, EU-, and ACR-thyroid imaging recording and data systems and ATA guidelines for the ultrasound risk stratification of thyroid nodules. Endocrine 67, 143–154 (2020). https://doi.org/10.1007/s12020-019-02134-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12020-019-02134-1