Skip to main content
Log in

Value of a deep learning-based algorithm for detecting Lung-RADS category 4 nodules on chest radiographs in a health checkup population: estimation of the sample size for a randomized controlled trial

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objective

To explore the value of a deep learning-based algorithm in detecting Lung CT Screening Reporting and Data System category 4 nodules on chest radiographs from an asymptomatic health checkup population.

Methods

Data from an annual retrospective cohort of individuals who underwent chest radiographs for health checkup purposes and chest CT scanning within 3 months were collected. Among 3073 individuals, 118 with category 4 nodules on CT were selected. A reader performance test was performed using those 118 radiographs and randomly selected 51 individuals without any nodules. Four radiologists independently evaluated the radiographs without and with the results of the algorithm; and sensitivities/specificities were compared. The sample size needed to confirm the difference in detection rates was calculated, i.e., the number of true-positive radiographs divided by the total number of radiographs.

Results

The sensitivity of the radiologists substantially increased aided by the algorithm (38.8% [183/472] to 45.1% [213/472]; p < .001) without significant change in specificity (94.1% [192/204] vs. 92.2% [188/204]; p = .22). Pooled radiologists detected more nodules with the algorithm (32.0% [156/488] vs. 38.9% [190/488]; p < .001), without alteration of false-positive rates (0.09 [62/676], both). Pooled detection rates for the annual cohort were 1.49% (183/12,292) and 1.73% (213/12,292) without and with the algorithm, respectively. A sample size of 41,776 in each arm would be required to demonstrate significant detection rate difference with < 5% type I error and > 80% power.

Conclusion

Although readers substantially increased sensitivity in detecting nodules on chest radiographs from a health checkup population aided by the algorithm, detection rate difference was only 0.24%, requiring a sample size >80,000 for a randomized controlled trial.

Key Points

• Aided by a deep learning algorithm, pooled radiologists improved their sensitivity in detecting Lung-RADS category 4 nodules on chest radiographs from a health checkup population (38.8% [183/472] to 45.1% [213/472]; p < .001), without increasing false-positive rate.

• The prevalence of the Lung-RADS category 4 nodules was 3.8% (118/3073) on the population, resulting in only 0.24% increase of the detection rate for the radiologists with assistance of the algorithm.

• To confirm the significant detection rate increase by a randomized controlled trial, a sample size of 84,000 would be required.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

AUFROC:

Area-under-the jackknife free-ROC curve

AUROC:

Area-under-the receiver-operating characteristic curve

JAFROC:

Jackknife alternative free-response receiver-operating characteristic

Lung-RADS:

Lung CT Screening Reporting and Data System

References

  1. Torre LA, Siegel RL, Jemal A (2016) Lung cancer statistics. Lung cancer and personalized medicine. Springer, pp 1–19

  2. National Lung Screening Trial Research Team, Aberle DR, Adams AM et al (2011) Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 365:395–409

  3. De Koning H, Van Der Aalst C, De Jong P et al (2020) Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med 382(6):503–513

  4. Oken MM, Hocking WG, Kvale PA et al (2011) Screening by chest radiograph and lung cancer mortality: the Prostate, Lung, Colorectal, and Ovarian (PLCO) randomized trial. JAMA 306:1865–1873

    Article  CAS  Google Scholar 

  5. Pastorino U, Silva M, Sestini S et al (2019) Prolonged lung cancer screening reduced 10-year mortality in the MILD trial: new confirmation of lung cancer screening efficacy. Ann Oncol 30:1162–1169

    Article  CAS  Google Scholar 

  6. Mettler Jr FA, Mahesh M, Bhargavan-Chatfield M (2020) Patient exposure from radiologic and nuclear medicine procedures in the United States: procedure volume and effective dose for the period 2006–2016. Radiology 295(2):418–427

  7. Nam JG, Park S, Hwang EJ et al (2019) Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 290:218–228

    Article  Google Scholar 

  8. Sim Y, Chung MJ, Kotter E et al (2020) Deep convolutional neural network–based software improves radiologist detection of malignant lung nodules on chest radiographs. Radiology 294:199–209

    Article  Google Scholar 

  9. Hwang EJ, Park CM (2020) Clinical implementation of deep learning in thoracic radiology: potential applications and challenges. Korean J Radiol 21:511–525

    Article  Google Scholar 

  10. American College of Radiology. Lung CT Screening Reporting and Data System (Lung-RADS). Available at: http://www.acr.org/Quality-Safety/Resources/LungRADS. Accessed 25 Oct 2020

  11. Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning–based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:e191095–e191095

    Article  Google Scholar 

  12. Chakraborty DP (2011) Recent developments in imaging system assessment methodology, FROC analysis and the search model. Nucl Instrum Methods Phys Res, Sect A 648:S297–S301

    Article  CAS  Google Scholar 

  13. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845

  14. Bland JM, Altman DG (1995) Multiple significance tests: the Bonferroni method. BMJ 310:170

    Article  CAS  Google Scholar 

  15. Rajpurkar P, Irvin J, Ball RL et al (2018) Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 15:e1002686

    Article  Google Scholar 

  16. Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491

    Article  Google Scholar 

  17. Li Q, Balagurunathan Y, Liu Y et al (2018) Comparison between radiological semantic features and lung-rads in predicting malignancy of screen-detected lung nodules in the National Lung Screening Trial. Clin Lung Cancer 19:148 156. e143

    Article  Google Scholar 

Download references

Funding

This study has received funding by a grant of the Korea Health Technology R & D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI19C1129, HI15C1532).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jin Mo Goo.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Jin Mo Goo.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Methodology

• Retrospective

• Observational

• Performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nam, J.G., Kim, H.J., Lee, E.H. et al. Value of a deep learning-based algorithm for detecting Lung-RADS category 4 nodules on chest radiographs in a health checkup population: estimation of the sample size for a randomized controlled trial. Eur Radiol 32, 213–222 (2022). https://doi.org/10.1007/s00330-021-08162-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-021-08162-8

Keywords

Navigation