Skip to main content

Advertisement

SpringerLink
Log in

Effectiveness of radiologist training in improving reader agreement for Lung-RADS 4X categorization

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To identify the agreement on Lung CT Screening Reporting and Data System 4X categorization between radiologists and an expert-adjudicated reference standard and to investigate whether training led to improvement of the agreement measures and diagnostic potential for lung cancer.

Methods

Category 4 nodules in the Korean Lung Cancer Screening Project were identified retrospectively, and each 4X nodule was matched with one 4A or 4B nodule. An expert panel re-evaluated the categories and determined the reference standard. Nineteen radiologists were asked to determine the presence of CT features of malignancy and 4X categorization for each nodule. A review was performed in two sessions, and training material was given after session 1. Agreement on 4X categorization between radiologists and the expert-adjudicated reference standard and agreement between radiologist-assessed 4X categorization and lung cancer diagnosis were evaluated.

Results

The 48 expert-adjudicated 4X nodules and 64 non-4X nodules were evenly distributed in each session. The proportion of category 4X decreased after training (56.4% ± 16.9% vs. 33.4% ± 8.0%; p < 0.001). Cohen’s κ indicated poor agreement (0.39 ± 0.16) in session 1, but agreement improved in session 2 (0.47 ± 0.09; p = 0.03). The increase in agreement in session 2 was observed among inexperienced radiologists (p < 0.05), and experienced and inexperienced reviewers exhibited comparable agreement performance in session 2 (p > 0.05). All agreement measures between radiologist-assessed 4X categorization and lung cancer diagnosis increased in session 2 (p < 0.05).

Conclusion

Radiologist training can improve reader agreement on 4X categorization, leading to enhanced diagnostic performance for lung cancer.

Key Points

• Agreement on 4X categorization between radiologists and an expert-adjudicated reference standard was initially poor, but improved significantly after training.

• The mean proportion of 4X categorization by 19 radiologists decreased from 56.4% ± 16.9% in session 1 to 33.4% ± 8.0% in session 2.

• All agreement measures between the 4X categorization and lung cancer diagnosis increased significantly in session 2, implying that appropriate training and guidance increased the diagnostic potential of category 4X.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

AC1 :

First-order agreement coefficient

K-LUCAS:

Korean Lung Cancer Screening Project

Lung-RADS:

Lung CT Screening Reporting and Data System

References

  1. de Koning HJ, van der Aalst CM, de Jong PA et al (2020) Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med 382:503–513

    Article  Google Scholar 

  2. National Lung Screening Trial Research Team, Aberle DR, Adams AM et al (2011) Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 365:395–409

    Article  Google Scholar 

  3. Pastorino U, Silva M, Sestini S et al (2019) Prolonged lung cancer screening reduced 10-year mortality in the MILD trial: new confirmation of lung cancer screening efficacy. Ann Oncol 30:1162–1169

    Article  CAS  Google Scholar 

  4. American College of Radiology (2014) Lung CT Screening Reporting & Data System (Lung-RADS) version 1.0. Available via https://www.acr.org/-/media/ACR/Files/RADS/Lung-RADS/LungRADS_AssessmentCategories.pdf?la=en. Accessed July 01 2019

  5. Mehta HJ, Mohammed TL, Jantz MA (2017) The American College of Radiology Lung Imaging Reporting and Data System: potential drawbacks and need for revision. Chest 151:539–543

    Article  Google Scholar 

  6. McKee BJ, Regis SM, McKee AB, Flacke S, Wald C (2015) Performance of ACR Lung-RADS in a clinical CT lung screening program. J Am Coll Radiol 12:273–276

    Article  Google Scholar 

  7. Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491

    Article  Google Scholar 

  8. van Riel SJ, Jacobs C, Scholten ET et al (2019) Observer variability for Lung-RADS categorisation of lung cancer screening CTs: impact on patient management. Eur Radiol 29:924–931

    Article  Google Scholar 

  9. Chung K, Jacobs C, Scholten ET et al (2017) Lung-RADS category 4X: does it improve prediction of malignancy in subsolid nodules? Radiology 284:264–271

    Article  Google Scholar 

  10. Kim H, Kim HY, Goo JM, Kim Y (2020) Lung cancer CT screening and Lung-RADS in a tuberculosis-endemic country: the Korean Lung Cancer Screening Project (K-LUCAS). Radiology 296:181–188

    Article  Google Scholar 

  11. Nair A, Bartlett EC, Walsh SLF et al (2018) Variable radiological lung nodule evaluation leads to divergent management recommendations. Eur Respir J 52:1801359

    Article  Google Scholar 

  12. Lee J, Lim J, Kim Y et al (2019) Development of protocol for Korean Lung Cancer Screening Project (K-LUCAS) to evaluate effectiveness and feasibility to implement national cancer screening program. Cancer Res Treat 51:1285–1294

    Article  Google Scholar 

  13. Hwang EJ, Goo JM, Kim HY, Yi J, Yoon SH, Kim Y (2021) Implementation of the cloud-based computerized interpretation system in a nationwide lung cancer screening with low-dose CT: comparison with the conventional reading system. Eur Radiol 31:475–485

    Article  Google Scholar 

  14. Suh YJ, Lee JW, Shin SY, Goo JM, Kim Y, Yong HS (2020) Coronary artery calcium severity grading on non-ECG-gated low-dose chest computed tomography: a multiple-observer study in a nationwide lung cancer screening registry. Eur Radiol 30:3684–3691

    Article  CAS  Google Scholar 

  15. Hwang EJ, Goo JM, Kim HY et al (2020) Variability in interpretation of low-dose chest CT using computerized assessment in a nationwide lung cancer screening program: comparison of prospective reading at individual institutions and retrospective central reading. Eur Radiol. https://doi.org/10.1007/s00330-020-07424-1

  16. Kim H, Kim HY, Goo JM, Kim Y (2020) External validation and comparison of the Brock model and Lung-RADS for the baseline lung cancer CT screening using data from the Korean Lung Cancer Screening Project. Eur Radiol. https://doi.org/10.1007/s00330-020-07513-1

  17. Martin MD, Kanne JP, Broderick LS, Kazerooni EA, Meyer CA (2017) Lung-RADS: pushing the limits. Radiographics 37:1975–1993

    Article  Google Scholar 

  18. McWilliams A, Tammemagi MC, Mayo JR et al (2013) Probability of cancer in pulmonary nodules detected on first screening CT. N Engl J Med 369:910–919

    Article  CAS  Google Scholar 

  19. Erasmus JJ, Connolly JE, McAdams HP, Roggli VL (2000) Solitary pulmonary nodules: part I. morphologic evaluation for differentiation of benign and malignant lesions. Radiographics 20:43–58

    Article  CAS  Google Scholar 

  20. Gurney JW (1993) Determining the likelihood of malignancy in solitary pulmonary nodules with Bayesian analysis. Part I. Theory. Radiology 186:405–413

    Article  CAS  Google Scholar 

  21. Zwirewich CV, Vedal S, Miller RR, Muller NL (1991) Solitary pulmonary nodule: high-resolution CT and radiologic-pathologic correlation. Radiology 179:469–476

    Article  CAS  Google Scholar 

  22. Kui M, Templeton PA, White CS, Cai ZL, Bai YX, Cai YQ (1996) Evaluation of the air bronchogram sign on CT in solitary pulmonary lesions. J Comput Assist Tomogr 20:983–986

    Article  CAS  Google Scholar 

  23. Fleiss JL, Levin B, Paik MC (2003) Statistical methods for rates and proportions, 3rd edn. Wiley, New York

    Book  Google Scholar 

  24. Giacalone M, Agata Z, Cozzucoli PC, Alibrandi A (2018) Bonferroni-Holm and permutation tests to compare health data: methodological and applicative issues. BMC Med Res Methodol 18:81

    Article  Google Scholar 

  25. Bender R, Lange S (2001) Adjusting for multiple testing--when and how? J Clin Epidemiol 54:343–349

    Article  CAS  Google Scholar 

  26. Henschke CI, Yankelevitz DF, Mirtcheva R et al (2002) CT screening for lung cancer: frequency and significance of part-solid and nonsolid nodules. AJR Am J Roentgenol 178:1053–1057

    Article  Google Scholar 

  27. Scholten ET, de Jong PA, de Hoop B et al (2015) Towards a close computed tomography monitoring approach for screen detected subsolid pulmonary nodules? Eur Respir J 45:765–773

    Article  Google Scholar 

  28. American College of Radiology (2019) ACR lung cancer screening webinars. Available via https://www.acr.org/Advocacy-and-Economics/Advocacy-News/Advocacy-News-Issues/In-the-August-31-2019-Issue/ACR-Lung-Cancer-Screening-Webinars-Now-Available. Accessed August 24 2020

  29. Pinsky PF, Church TR, Izmirlian G, Kramer BS (2013) The National Lung Screening Trial: results stratified by demographics, smoking history, and lung cancer histology. Cancer 119:3976–3983

    Article  Google Scholar 

Download references

Funding

This study was supported by grants from the National R&D Program for Cancer Control, Ministry of Health and Welfare (1720310, 1520230) and National Health Promotion Fund (1760810-1), Ministry of Health and Welfare, Republic of Korea.

Author information

Authors and Affiliations

  1. Department of Radiology, Seoul National University College of Medicine, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Hyungjin Kim, Jin Mo Goo, Junghoan Park, Juil Park, Jiseon Oh & Woo Hyeon Lim

  2. Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, South Korea

    Hyungjin Kim & Jin Mo Goo

  3. Cancer Research Institute, Seoul National University, Seoul, South Korea

    Jin Mo Goo

  4. Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea

    Tae Jung Kim

  5. National Cancer Center, Goyang, South Korea

    Hyae Young Kim & Yeol Kim

  6. Pohang St. Mary’s Hospital, Pohang, South Korea

    Guanmin Gu

  7. Bucheon St. Mary’s Hospital, The Catholic University of Korea, Seoul, South Korea

    Bomi Gil

  8. Asan Medical Center, Seoul, South Korea

    Wooil Kim

  9. Dankook University Hospital, Cheonan, South Korea

    Seon Young Park

  10. Design Hospital, Jeonju, South Korea

    Harkhoon Park

  11. Jeonju Hospital, Jeonju, South Korea

    Wonkyu Song

  12. SoonChunHyang University Bucheon Hospital, Bucheon, South Korea

    Kyung Eun Shin

  13. Seoul National University Bundang Hospital, Seongnam, South Korea

    Sung Hyun Yoon

  14. Lunit Inc., Seoul, South Korea

    Sanghyup Lee

  15. Hanyang University Guri Hospital, Guri, South Korea

    Youkyung Lee

  16. Chonnam National University Hwasun Hospital, Hwasun-gun, South Korea

    Won Gi Jeong

  17. Seoul St. Mary’s Hospital, The Catholic University of Korea, Seoul, South Korea

    Jung Im Jung

  18. Chung-Ang University Hospital, Seoul, South Korea

    Min Jae Cha

  19. Wesarang Internal Medicine Clinic, Jeonju, South Korea

    Sinae Choi

  20. Korean Armed Forces Capital Hospital, Seongnam, South Korea

    Hyoung In Choi

  21. Sungkyunkwan University Kangbuk Samsung Hospital, Seoul, South Korea

    Soo-Youn Ham

Authors
  1. Hyungjin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin Mo Goo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tae Jung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hyae Young Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guanmin Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bomi Gil
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wooil Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Seon Young Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Junghoan Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Juil Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Harkhoon Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Wonkyu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Kyung Eun Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Jiseon Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Sung Hyun Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Sanghyup Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Youkyung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Woo Hyeon Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Won Gi Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Jung Im Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Min Jae Cha
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Sinae Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Hyoung In Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Soo-Youn Ham
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Yeol Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Mo Goo.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Jin Mo Goo.

Conflict of interest

Activities related to the present article: none.

Activities not related to the present article: HK received a research grant from Lunit; holds stock in Medical IP. JMG received research grants from Lunit, INFINITT Healthcare, Dongkook Lifescience, and LG electronics.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was not required because this study retrospectively analyzed anonymized data, which was archived from a prospective cohort.

Ethical approval

An exemption from the institutional review board of Seoul National University Hospital was acquired for this retrospective analysis (E-2008-157-1151).

Study subjects or cohorts overlap

Some study subjects or cohorts have been previously reported

Radiology 2020;296(1):181-188;

Eur Radiol 2020;30(7):3684-3691;

Eur Radiol doi: 10.1007/s00330-020-07151-7;

Eur Radiol doi: 10.1007/s00330-020-07424-1

Methodology

• retrospective

• diagnostic or prognostic study

• multicenter study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

ESM 1

(DOCX 507 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, H., Goo, J.M., Kim, T.J. et al. Effectiveness of radiologist training in improving reader agreement for Lung-RADS 4X categorization. Eur Radiol 31, 8147–8159 (2021). https://doi.org/10.1007/s00330-021-07990-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-021-07990-y

Keywords

Search

Navigation