Skip to main content
SpringerLink
Log in

External validation and comparison of the Brock model and Lung-RADS for the baseline lung cancer CT screening using data from the Korean Lung Cancer Screening Project

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To validate and compare the performance of the Brock model and Lung CT Screening Reporting and Data System (Lung-RADS) on nodules detected by baseline CT screening.

Methods

We performed a secondary analysis of the Korean Lung Cancer Screening Project (K-LUCAS; ClinicalTrials.gov, NCT03394703), a nationwide, multicenter, prospective cohort study. From April 2017 to December 2018, low-dose CT screening was performed on high-risk subjects. Discrimination and calibration of Brock models 2a and 2b (i.e., full model without and with spiculation, respectively) were assessed, and discrimination was compared with that of Lung-RADS, which utilized subjective assessment categories 2b (b stands for benign) and 4X.

Results

Of the 13,150 subjects, 4578 were eligible (median age 62 years; 4458 men; 9929 nodules including 40 lung cancers). Areas under the receiver operating characteristic curve were 0.96 (IQR 0.92–0.99) for Brock model 2a, 0.96 (IQR 0.92–0.99) for Brock model 2b, and 0.95 (IQR 0.91–0.99) for Lung-RADS (p = 0.32 and p = 0.34, respectively). At an equivalent cutoff of 5%, Brock model 2b (sensitivity 87.5% [35/40]; specificity 93.6% [9259/9889]; positive predictive value [PPV] 5.3% [35/665]; negative predictive value [NPV] 99.9% [9259/9264]) and Lung-RADS (sensitivity 87.5% [35/40]; specificity 93.3% [9222/9889]; PPV 5.0% [35/702]; NPV 99.9% [9222/9227]) performed similarly well (all p > 0.05). The calibration performance of both Brock models 2a and 2b was poor (both p < 0.001).

Conclusions

Lung-RADS, when reinforced with visual assessment–based categories, has a similar diagnostic performance to the Brock model for baseline CT scans.

Key Points

• Brock model 2b and Lung CT Screening Reporting and Data System (Lung-RADS) demonstrated a similar discrimination performance for lung cancer in the baseline CT screening (areas under the receiver operating characteristic curve 0.96 vs. 0.95; p = 0.34).

• When visual assessment–based categories were removed from Lung-RADS, specificity and positive predictive value were lower than those of Brock model 2b (p = 0.001 and p = 0.02, respectively).

• The Brock model showed poor calibration (p < 0.001).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Abbreviations

AUC:

Area under the receiver operating characteristic curve

IQR:

Interquartile range

K-LUCAS:

Korean Lung Cancer Screening Project

Lung-RADS:

Lung CT Screening Reporting and Data System

NLST:

National Lung Screening Trial

NPV:

Negative predictive value

PPV:

Positive predictive value

References

  1. National Lung Screening Trial Research Team, Aberle DR, Adams AM et al (2011) Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 365:395–409

    Article  Google Scholar 

  2. de Koning HJ, van der Aalst CM, de Jong PA et al (2020) Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med 382:503–513

    Article  Google Scholar 

  3. Pastorino U, Silva M, Sestini S et al (2019) Prolonged lung cancer screening reduced 10-year mortality in the MILD trial: new confirmation of lung cancer screening efficacy. Ann Oncol 30:1162–1169

    Article  CAS  Google Scholar 

  4. American College of Radiology (2014) Lung CT Screening Reporting & Data System (Lung-RADS) version 1.0. Available via https://www.acr.org/-/media/ACR/Files/RADS/Lung-RADS/LungRADS_AssessmentCategories.pdf?la=en. Accessed 1 July 2019

  5. Martin MD, Kanne JP, Broderick LS, Kazerooni EA, Meyer CA (2017) Lung-RADS: pushing the limits. Radiographics 37:1975–1993

    Article  Google Scholar 

  6. Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491

    Article  Google Scholar 

  7. Mehta HJ, Mohammed TL, Jantz MA (2017) The American College of Radiology Lung Imaging Reporting and Data System: potential drawbacks and need for revision. Chest 151:539–543

    Article  Google Scholar 

  8. McWilliams A, Tammemagi MC, Mayo JR et al (2013) Probability of cancer in pulmonary nodules detected on first screening CT. N Engl J Med 369:910–919

    Article  CAS  Google Scholar 

  9. White CS, Dharaiya E, Campbell E, Boroczky L (2017) The Vancouver Lung Cancer Risk Prediction Model: assessment by using a subset of the National Lung Screening Trial cohort. Radiology 283:264–272

    Article  Google Scholar 

  10. Winkler Wille MM, van Riel SJ, Saghir Z et al (2015) Predictive accuracy of the PanCan lung cancer risk prediction model - external validation based on CT from the Danish Lung Cancer Screening Trial. Eur Radiol 25:3093–3099

    Article  Google Scholar 

  11. White CS, Dharaiya E, Dalal S, Chen R, Haramati LB (2019) Vancouver risk calculator compared with ACR Lung-RADS in predicting malignancy: analysis of the National Lung Screening Trial. Radiology 291:205–211

    Article  Google Scholar 

  12. van Riel SJ, Ciompi F, Jacobs C et al (2017) Malignancy risk estimation of screen-detected nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN guidelines. Eur Radiol 27:4019–4029

    Article  Google Scholar 

  13. National Lung Screening Trial Research Team, Aberle DR, Adams AM et al (2010) Baseline characteristics of participants in the randomized national lung screening trial. J Natl Cancer Inst 102:1771–1779

    Article  Google Scholar 

  14. Lee J, Lim J, Kim Y et al (2019) Development of protocol for Korean Lung Cancer Screening Project (K-LUCAS) to evaluate effectiveness and feasibility to implement national cancer screening program. Cancer Res Treat 51:1285–1294

    Article  Google Scholar 

  15. Burrill J, Williams CJ, Bain G, Conder G, Hine AL, Misra RR (2007) Tuberculosis: a radiologic review. Radiographics 27:1255–1273

    Article  Google Scholar 

  16. Kim H, Kim HY, Goo JM, Kim Y (2020) Lung cancer CT screening and Lung-RADS in a tuberculosis-endemic country: the Korean Lung Cancer Screening Project (K-LUCAS). Radiology 296:181–188

    Article  Google Scholar 

  17. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845

    Article  CAS  Google Scholar 

  18. McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157

    Article  CAS  Google Scholar 

  19. Leisenring W, Alonzo T, Pepe MS (2000) Comparisons of predictive values of binary medical diagnostic tests for paired designs. Biometrics 56:345–351

    Article  CAS  Google Scholar 

  20. Tammemagi MC, Ten Haaf K, Toumazis I et al (2019) Development and validation of a multivariable lung cancer risk prediction model that includes low-dose computed tomography screening results: a secondary analysis of data from the National Lung Screening Trial. JAMA Netw Open 2:e190204

    Article  Google Scholar 

  21. Walsh CG, Sharman K, Hripcsak G (2017) Beyond discrimination: a comparison of calibration methods and clinical usefulness of predictive models of readmission risk. J Biomed Inform 76:9–18

    Article  Google Scholar 

  22. Kim HY, Song KS, Goo JM, Lee JS, Lee KS, Lim TH (2001) Thoracic sequelae and complications of tuberculosis. Radiographics 21:839–858

    Article  CAS  Google Scholar 

  23. Shipe ME, Deppen SA, Farjah F, Grogan EL (2019) Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis 11:S574–S584

    Article  Google Scholar 

  24. Winter A, Aberle DR, Hsu W (2019) External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data. Thorax 74:551–563

    Article  Google Scholar 

  25. International Agency For Research on Cancer (2020) Data visualization tools for exploring the global cancer burden in 2018. Available via https://gco.iarc.fr/today/home. Accessed 30 Jan 2020

Download references

Funding

This study was supported by grants from the National R&D Program for Cancer Control, Ministry of Health and Welfare (1720310, 1520230), and the National Health Promotion Fund (1760810-1), Ministry of Health and Welfare, Republic of Korea.

Author information

Authors and Affiliations

  1. Department of Radiology, Seoul National University Hospital, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Hyungjin Kim & Jin Mo Goo

  2. Institute of Radiation Medicine, Seoul National University Medical Research Center, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Hyungjin Kim & Jin Mo Goo

  3. Department of Diagnostic Radiology, National Cancer Center, 323, Ilsan-ro, Ilsandong-gu, Goyang, 10408, South Korea

    Hyae Young Kim

  4. Cancer Research Institute, Seoul National University, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Jin Mo Goo

  5. Cancer Early Detection Branch, National Cancer Control Institute, National Cancer Center, 323, Ilsan-ro, Ilsandong-gu, Goyang, 10408, South Korea

    Yeol Kim

Authors
  1. Hyungjin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyae Young Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Mo Goo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yeol Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyae Young Kim.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Hyae Young Kim.

Conflict of interest

Activities related to the present article: none.

Activities not related to the present article: HK received a research grant from Lunit. JMG received research grants from Lunit, Infinitt Healthcare, and Dongkook Lifescience.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

This study was approved by the institutional review boards of the participating medical centers in South Korea, which were National Cancer Center, Daegu-Gyeongbuk Regional Cancer Center, Daejeon Regional Cancer Center, Gangwon Cancer Center, Incheon Regional Cancer Center, Jeonbuk Regional Cancer Center, Jeju Regional Cancer Center, Jeonnam Regional Cancer Center, Gyeonggi Cancer Center, Kyunghee University Hospital, Korea University Guro Hospital, Busan Regional Cancer Center, Seoul National University Hospital, and Ulsan Cancer Center (representative IRB No. NCC2017-0078).

Study subjects or cohorts overlap

Some study subjects or cohorts have been previously reported (Radiology 10.1148/radiol.2020192283:192283; Eur Radiol 10.1007/s00330-020-06707-x; Unpublished work by Hwang et al [submitted state]).

Methodology

• retrospective

• diagnostic or prognostic study

• multicenter study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 34 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, H., Kim, H.Y., Goo, J.M. et al. External validation and comparison of the Brock model and Lung-RADS for the baseline lung cancer CT screening using data from the Korean Lung Cancer Screening Project. Eur Radiol 31, 4004–4015 (2021). https://doi.org/10.1007/s00330-020-07513-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-020-07513-1

Keywords

Search

Navigation