Skip to main content

Deep learning–based automated detection algorithm for active pulmonary tuberculosis on chest radiographs: diagnostic performance in systematic screening of asymptomatic individuals

European Radiology volume 31, pages 1069–1080 (2021)Cite this article

Abstract

Objectives

Performance of deep learning–based automated detection (DLAD) algorithms in systematic screening for active pulmonary tuberculosis is unknown. We aimed to validate DLAD algorithm for detection of active pulmonary tuberculosis and any radiologically identifiable relevant abnormality on chest radiographs (CRs) in this setting.

Methods

We performed out-of-sample testing of a pre-trained DLAD algorithm, using CRs from 19.686 asymptomatic individuals (ages, 21.3 ± 1.9 years) as part of systematic screening for tuberculosis between January 2013 and July 2018. Area under the receiver operating characteristic curves (AUC) for diagnosis of tuberculosis and any relevant abnormalities were measured. Accuracy measures including sensitivities, specificities, positive predictive values (PPVs), and negative predictive values (NPVs) were calculated at pre-defined operating thresholds (high sensitivity threshold, 0.16; high specificity threshold, 0.46).

Results

All five CRs from four individuals with active pulmonary tuberculosis were correctly classified as having abnormal findings by DLAD with specificities of 0.959 and 0.997, PPVs of 0.006 and 0.068, and NPVs of both 1.000 at high sensitivity and high specificity thresholds, respectively. With high specificity thresholds, DLAD showed comparable diagnostic measures with the pooled radiologists (p values > 0.05). For the radiologically identifiable relevant abnormality (n = 28), DLAD showed an AUC value of 0.967 (95% confidence interval, 0.938–0.996) with sensitivities of 0.821 and 0.679, specificities of 0.960 and 0.997, PPVs of 0.028 and 0.257, and NPVs of both 0.999 at high sensitivity and high specificity thresholds, respectively.

Conclusions

In systematic screening for tuberculosis in a low-prevalence setting, DLAD algorithm demonstrated excellent diagnostic performance, comparable with the radiologists in the detection of active pulmonary tuberculosis.

Key Points

• Deep learning–based automated detection algorithm detected all chest radiographs with active pulmonary tuberculosis with high specificities and negative predictive values in systematic screening.

• Deep learning–based automated detection algorithm had comparable diagnostic measures with the radiologists for detection of active pulmonary tuberculosis on chest radiographs.

• For the detection of radiologically identifiable relevant abnormalities on chest radiographs, deep learning–based automated detection algorithm showed excellent diagnostic performance in systematic screening.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Abbreviations

CRs:

Chest radiographs

DLAD:

Deep learning–based automated detection

TB:

Tuberculosis

References

  1. World Health Organization (2019) Global tuberculosis repot 2019. World Health Organization, Geneva. Available via https://www.who.int/tb/publications/global_report/en/. Accessed 12 Feb 2020

  2. World Health Organization (2016) Chest radiography in tuberculosis detection: summary of current WHO recommendations and guidance on programmatic approaches. World Health Organization, Geneva Available via https://www.who.int/tb/publications/chest-radiography/en/. Accessed 12 Feb 2020

    Google Scholar 

  3. World Health Organization (2013) Systematic screening for active tuberculosis: principles and recommendations. World Health Organization, Geneva Available via https://www.who.int/tb/tbscreening/en/. Accessed 12 Feb 2020

    Google Scholar 

  4. World Health Organization (2011) Early detection of tuberculosis: an overview of approaches, guidelines and tools. World Health Organization, Geneva Available via https://apps.who.int/iris/handle/10665/70824. Accessed 12 Feb 2020

    Google Scholar 

  5. World Health Organization (2015) The global plan to stop TB, 2016-2020. World Health Organization, Geneva. Available via http://www.stoptb.org/global/plan/plan2/. Accessed 12 Feb 2020

  6. World Health Organization (2010) Public-private mix for TB care and control. World Health Organization, Geneva Available via https://www.who.int/tb/publications/tb-publicprivate-toolkit/en/. Accessed 12 Feb 2020

    Google Scholar 

  7. Yoon C, Dowdy DW, Esmail H, MacPherson P, Schumacher SG (2019) Screening for tuberculosis: time to move beyond symptoms. Lancet Respir Med 7:202–204

    Article  Google Scholar 

  8. Dara M, Solovic I, Sotgiu G et al (2016) Tuberculosis care among refugees arriving in Europe: a ERS/WHO Europe Region survey of current practices. Eur Respir J 48:808–817

    Article  Google Scholar 

  9. Melendez J, Sánchez CI, Philipsen RH et al (2016) An automated tuberculosis screening strategy combining X-ray-based computer-aided detection and clinical information. Sci Rep 6:25265

    CAS  Article  Google Scholar 

  10. Pande T, Cohen C, Pai M, Ahmad Khan F (2016) Computer-aided detection of pulmonary tuberculosis on digital chest radiographs: a systematic review. Int J Tuberc Lung Dis 20:1226–1230

    CAS  Article  Google Scholar 

  11. Hogeweg L, Mol C, de Jong PA, Dawson R, Ayles H, van Ginneken B (2010) Fusion of local and global detection systems to detect tuberculosis in chest radiographs. Med Image Comput Comput Assist Interv 13:650–657

    PubMed  Google Scholar 

  12. Rahman MT, Codlin AJ, Rahman MM et al (2017) An evaluation of automated chest radiography reading software for tuberculosis screening among public-and private-sector patients. Eur Respir J 49:1602159

    Article  Google Scholar 

  13. Jaeger S, Karargyris A, Candemir S et al (2014) Automatic tuberculosis screening using chest radiographs. IEEE Trans Med Imaging 33:233–245

    Article  Google Scholar 

  14. Nam JG, Park S, Hwang EJ et al (2018) Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 290:218–228

    Article  Google Scholar 

  15. Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automatic detection algorithm for active pulmonary tuberculosis on chest radiographs. Clin Infect Dis 69:739–747

    Article  Google Scholar 

  16. Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:3191095

    Article  Google Scholar 

  17. Qin ZZ, Sander MS, Rai B et al (2019) Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems. Sci Rep 9:15000

    Article  Google Scholar 

  18. Go U, Park M, Kim U-N et al (2018) Tuberculosis prevention and care in Korea: evolution of policy and practice. J Clin Tuberc Other Mycobact Dis 11:28–36

    Article  Google Scholar 

  19. (2000) Diagnostic Standards and Classification of Tuberculosis in Adults and Children. This official statement of the American Thoracic Society and the Centers for Disease Control and Prevention was adopted by the ATS Board of Directors, July 1999. This statement was endorsed by the Council of the Infectious Disease Society of America, September 1999. Am J Respir Crit Care Med 161:1376–1395

  20. Skoura E, Zumla A, Bomanji J (2015) Imaging in tuberculosis. Int J Infect Dis 32:87–93

    Article  Google Scholar 

  21. Geng E, Kreiswirth B, Burzynski J, Schluger NW (2005) Clinical and radiographic correlates of primary and reactivation tuberculosis: a molecular epidemiology study. JAMA 293:2740–2745

    CAS  Article  Google Scholar 

  22. Moskowitz CS, Pepe MS (2006) Comparing the predictive values of diagnostic tests: sample size and analysis for paired study designs. Clin Trials 3:272–279

    Article  Google Scholar 

  23. Lakhani P, Sundaram B (2017) Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284:574–582

    Article  Google Scholar 

  24. Philipsen R, Sánchez C, Maduskar P et al (2015) Automated chest-radiography as a triage for Xpert testing in resource-constrained settings: a prospective study of diagnostic accuracy and costs. Sci Rep 5:12215

    CAS  Article  Google Scholar 

  25. Ting DSW, Tan T-E, Lim C (2019) Development and validation of a deep learning system for detection of active pulmonary tuberculosis on chest radiographs: clinical and technical considerations. Clin Infect Dis 69:748–750

    Article  Google Scholar 

  26. Park SH (2019) Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance. Radiology 290:272–372

    Article  Google Scholar 

  27. Ting DS, Yi PH, Hui F (2018) Clinical applicability of deep learning system in detecting tuberculosis with chest radiography. Radiology 286:729–731

    Article  Google Scholar 

  28. Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800–809

    Article  Google Scholar 

  29. Rutjes AW, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PM (2005) Case–control and two-gate designs in diagnostic accuracy studies. Clin Chem 126:1777–1782

    Google Scholar 

  30. LoBue PA, Moser KS (2004) Screening of immigrants and refugees for pulmonary tuberculosis in San Diego County, California. Chest 126:1777–1782

    Article  Google Scholar 

  31. Wang PD, Lin RS (2000) Tuberculosis transmission in the family. J Infect 41:249–251

    CAS  Article  Google Scholar 

  32. Ross J, Reid K, Jamieson A (1977) Pulmonary tuberculosis in the common hostel population. Update 7:167–174

    Google Scholar 

  33. Den Boon S, Verver S, Lombard C et al (2008) Comparison of symptoms and treatment outcomes between actively and passively detected tuberculosis cases: the additional value of active case finding. Epidemiol Infect 136:1342–1349

    Article  Google Scholar 

  34. Eang MT, Satha P, Yadav RP et al (2012) Early detection of tuberculosis through community-based active case finding in Cambodia. BMC Public Health 12:469

    Article  Google Scholar 

  35. Santha T, Renu G, Frieden T et al (2003) Are community surveys to detect tuberculosis in high prevalence areas useful? Results of a comparative study from Tiruvallur District, South India. Int J Tuberc Lung Dis 7:258–265

    CAS  PubMed  Google Scholar 

  36. Parikh R, Mathai A, Parikh S, Sekhar GC, Thomas R (2008) Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol 56:45–50

    Article  Google Scholar 

Download references

Acknowledgments

This study was supported by the Seoul Research & Business Development Program (grant number FI170002), and Lunit Inc. provided technical supports for this study. There is a major research agreement between Seoul National University Hospital and Lunit Inc., in which the roles of researchers and Lunit Inc. were described. However, the funder and Lunit Inc. did not have any role either in the study design; in the collection, analysis, and interpretation of the data; in the writing of the report; and in the decision to submit the article for publication. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Researchers (J.H.L., E.J.H., W.Y.L., S.L., J.R.A.) who controlled, manipulated, and analyzed data did not have any conflict of interest. Three authors (J.M.G., H.K., C.M.P.) received research grants from Lunit Inc. for outside of this study. The authors state that this work has not received any funding.

Funding

This study was supported by the Seoul Research & Business Development Program (grant number FI170002).

Author information

Authors and Affiliations

  1. Armed Forces Seoul District Hospital, Seoul, South Korea

    Jong Hyuk Lee, Woo Young Lee & Sangho Lee

  2. Lunit Inc., Seoul, South Korea

    Sunggyun Park

  3. Department of Radiology, Seoul National University College of Medicine, Seoul, South Korea

    Eui Jin Hwang, Jin Mo Goo, Hyungjin Kim & Chang Min Park

  4. Airforce Surgeon General Office, Gyeryong-si, Chungcheongnam-do, South Korea

    Woo Young Lee

  5. Armed Forces Capital Hospital, Seongnam-si, Gyeonggi-do, South Korea

    Sangho Lee

  6. Division of Infectious Diseases and Geographic Medicine, Stanford University School of Medicine, Stanford, CA, USA

    Jason R. Andrews

Authors
  1. Jong Hyuk Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sunggyun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eui Jin Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin Mo Goo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Woo Young Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sangho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hyungjin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jason R. Andrews
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Chang Min Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chang Min Park.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Chang Min Park.

Conflict of interest

The authors of this manuscript declare relationships with the following companies: Three authors (J.M.G., H.K., C.M.P.) received research grants from Lunit Inc. for outside of this study. The authors state that this work has not received any funding.

Statistics and biometry

Two of the authors has significant statistical expertise (Chang Min Park and Jong Hyuk Lee).

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Methodology

• retrospective

• diagnostic study

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, J.H., Park, S., Hwang, E.J. et al. Deep learning–based automated detection algorithm for active pulmonary tuberculosis on chest radiographs: diagnostic performance in systematic screening of asymptomatic individuals. Eur Radiol 31, 1069–1080 (2021). https://doi.org/10.1007/s00330-020-07219-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-020-07219-4

Keywords

  • Tuberculosis
  • Mass screening
  • Radiography
  • Deep learning
  • Diagnosis, computer-assisted