Performance of deep learning–based automated detection (DLAD) algorithms in systematic screening for active pulmonary tuberculosis is unknown. We aimed to validate DLAD algorithm for detection of active pulmonary tuberculosis and any radiologically identifiable relevant abnormality on chest radiographs (CRs) in this setting.
We performed out-of-sample testing of a pre-trained DLAD algorithm, using CRs from 19.686 asymptomatic individuals (ages, 21.3 ± 1.9 years) as part of systematic screening for tuberculosis between January 2013 and July 2018. Area under the receiver operating characteristic curves (AUC) for diagnosis of tuberculosis and any relevant abnormalities were measured. Accuracy measures including sensitivities, specificities, positive predictive values (PPVs), and negative predictive values (NPVs) were calculated at pre-defined operating thresholds (high sensitivity threshold, 0.16; high specificity threshold, 0.46).
All five CRs from four individuals with active pulmonary tuberculosis were correctly classified as having abnormal findings by DLAD with specificities of 0.959 and 0.997, PPVs of 0.006 and 0.068, and NPVs of both 1.000 at high sensitivity and high specificity thresholds, respectively. With high specificity thresholds, DLAD showed comparable diagnostic measures with the pooled radiologists (p values > 0.05). For the radiologically identifiable relevant abnormality (n = 28), DLAD showed an AUC value of 0.967 (95% confidence interval, 0.938–0.996) with sensitivities of 0.821 and 0.679, specificities of 0.960 and 0.997, PPVs of 0.028 and 0.257, and NPVs of both 0.999 at high sensitivity and high specificity thresholds, respectively.
In systematic screening for tuberculosis in a low-prevalence setting, DLAD algorithm demonstrated excellent diagnostic performance, comparable with the radiologists in the detection of active pulmonary tuberculosis.
• Deep learning–based automated detection algorithm detected all chest radiographs with active pulmonary tuberculosis with high specificities and negative predictive values in systematic screening.
• Deep learning–based automated detection algorithm had comparable diagnostic measures with the radiologists for detection of active pulmonary tuberculosis on chest radiographs.
• For the detection of radiologically identifiable relevant abnormalities on chest radiographs, deep learning–based automated detection algorithm showed excellent diagnostic performance in systematic screening.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Deep learning–based automated detection
World Health Organization (2019) Global tuberculosis repot 2019. World Health Organization, Geneva. Available via https://www.who.int/tb/publications/global_report/en/. Accessed 12 Feb 2020
World Health Organization (2016) Chest radiography in tuberculosis detection: summary of current WHO recommendations and guidance on programmatic approaches. World Health Organization, Geneva Available via https://www.who.int/tb/publications/chest-radiography/en/. Accessed 12 Feb 2020
World Health Organization (2013) Systematic screening for active tuberculosis: principles and recommendations. World Health Organization, Geneva Available via https://www.who.int/tb/tbscreening/en/. Accessed 12 Feb 2020
World Health Organization (2011) Early detection of tuberculosis: an overview of approaches, guidelines and tools. World Health Organization, Geneva Available via https://apps.who.int/iris/handle/10665/70824. Accessed 12 Feb 2020
World Health Organization (2015) The global plan to stop TB, 2016-2020. World Health Organization, Geneva. Available via http://www.stoptb.org/global/plan/plan2/. Accessed 12 Feb 2020
World Health Organization (2010) Public-private mix for TB care and control. World Health Organization, Geneva Available via https://www.who.int/tb/publications/tb-publicprivate-toolkit/en/. Accessed 12 Feb 2020
Yoon C, Dowdy DW, Esmail H, MacPherson P, Schumacher SG (2019) Screening for tuberculosis: time to move beyond symptoms. Lancet Respir Med 7:202–204
Dara M, Solovic I, Sotgiu G et al (2016) Tuberculosis care among refugees arriving in Europe: a ERS/WHO Europe Region survey of current practices. Eur Respir J 48:808–817
Melendez J, Sánchez CI, Philipsen RH et al (2016) An automated tuberculosis screening strategy combining X-ray-based computer-aided detection and clinical information. Sci Rep 6:25265
Pande T, Cohen C, Pai M, Ahmad Khan F (2016) Computer-aided detection of pulmonary tuberculosis on digital chest radiographs: a systematic review. Int J Tuberc Lung Dis 20:1226–1230
Hogeweg L, Mol C, de Jong PA, Dawson R, Ayles H, van Ginneken B (2010) Fusion of local and global detection systems to detect tuberculosis in chest radiographs. Med Image Comput Comput Assist Interv 13:650–657
Rahman MT, Codlin AJ, Rahman MM et al (2017) An evaluation of automated chest radiography reading software for tuberculosis screening among public-and private-sector patients. Eur Respir J 49:1602159
Jaeger S, Karargyris A, Candemir S et al (2014) Automatic tuberculosis screening using chest radiographs. IEEE Trans Med Imaging 33:233–245
Nam JG, Park S, Hwang EJ et al (2018) Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 290:218–228
Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automatic detection algorithm for active pulmonary tuberculosis on chest radiographs. Clin Infect Dis 69:739–747
Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:3191095
Qin ZZ, Sander MS, Rai B et al (2019) Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems. Sci Rep 9:15000
Go U, Park M, Kim U-N et al (2018) Tuberculosis prevention and care in Korea: evolution of policy and practice. J Clin Tuberc Other Mycobact Dis 11:28–36
(2000) Diagnostic Standards and Classification of Tuberculosis in Adults and Children. This official statement of the American Thoracic Society and the Centers for Disease Control and Prevention was adopted by the ATS Board of Directors, July 1999. This statement was endorsed by the Council of the Infectious Disease Society of America, September 1999. Am J Respir Crit Care Med 161:1376–1395
Skoura E, Zumla A, Bomanji J (2015) Imaging in tuberculosis. Int J Infect Dis 32:87–93
Geng E, Kreiswirth B, Burzynski J, Schluger NW (2005) Clinical and radiographic correlates of primary and reactivation tuberculosis: a molecular epidemiology study. JAMA 293:2740–2745
Moskowitz CS, Pepe MS (2006) Comparing the predictive values of diagnostic tests: sample size and analysis for paired study designs. Clin Trials 3:272–279
Lakhani P, Sundaram B (2017) Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284:574–582
Philipsen R, Sánchez C, Maduskar P et al (2015) Automated chest-radiography as a triage for Xpert testing in resource-constrained settings: a prospective study of diagnostic accuracy and costs. Sci Rep 5:12215
Ting DSW, Tan T-E, Lim C (2019) Development and validation of a deep learning system for detection of active pulmonary tuberculosis on chest radiographs: clinical and technical considerations. Clin Infect Dis 69:748–750
Park SH (2019) Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance. Radiology 290:272–372
Ting DS, Yi PH, Hui F (2018) Clinical applicability of deep learning system in detecting tuberculosis with chest radiography. Radiology 286:729–731
Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800–809
Rutjes AW, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PM (2005) Case–control and two-gate designs in diagnostic accuracy studies. Clin Chem 126:1777–1782
LoBue PA, Moser KS (2004) Screening of immigrants and refugees for pulmonary tuberculosis in San Diego County, California. Chest 126:1777–1782
Wang PD, Lin RS (2000) Tuberculosis transmission in the family. J Infect 41:249–251
Ross J, Reid K, Jamieson A (1977) Pulmonary tuberculosis in the common hostel population. Update 7:167–174
Den Boon S, Verver S, Lombard C et al (2008) Comparison of symptoms and treatment outcomes between actively and passively detected tuberculosis cases: the additional value of active case finding. Epidemiol Infect 136:1342–1349
Eang MT, Satha P, Yadav RP et al (2012) Early detection of tuberculosis through community-based active case finding in Cambodia. BMC Public Health 12:469
Santha T, Renu G, Frieden T et al (2003) Are community surveys to detect tuberculosis in high prevalence areas useful? Results of a comparative study from Tiruvallur District, South India. Int J Tuberc Lung Dis 7:258–265
Parikh R, Mathai A, Parikh S, Sekhar GC, Thomas R (2008) Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol 56:45–50
This study was supported by the Seoul Research & Business Development Program (grant number FI170002), and Lunit Inc. provided technical supports for this study. There is a major research agreement between Seoul National University Hospital and Lunit Inc., in which the roles of researchers and Lunit Inc. were described. However, the funder and Lunit Inc. did not have any role either in the study design; in the collection, analysis, and interpretation of the data; in the writing of the report; and in the decision to submit the article for publication. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Researchers (J.H.L., E.J.H., W.Y.L., S.L., J.R.A.) who controlled, manipulated, and analyzed data did not have any conflict of interest. Three authors (J.M.G., H.K., C.M.P.) received research grants from Lunit Inc. for outside of this study. The authors state that this work has not received any funding.
This study was supported by the Seoul Research & Business Development Program (grant number FI170002).
The scientific guarantor of this publication is Chang Min Park.
Conflict of interest
The authors of this manuscript declare relationships with the following companies: Three authors (J.M.G., H.K., C.M.P.) received research grants from Lunit Inc. for outside of this study. The authors state that this work has not received any funding.
Statistics and biometry
Two of the authors has significant statistical expertise (Chang Min Park and Jong Hyuk Lee).
Written informed consent was waived by the Institutional Review Board.
Institutional Review Board approval was obtained.
• diagnostic study
• performed at one institution
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Lee, J.H., Park, S., Hwang, E.J. et al. Deep learning–based automated detection algorithm for active pulmonary tuberculosis on chest radiographs: diagnostic performance in systematic screening of asymptomatic individuals. Eur Radiol 31, 1069–1080 (2021). https://doi.org/10.1007/s00330-020-07219-4
- Mass screening
- Deep learning
- Diagnosis, computer-assisted