Skip to main content
SpringerLink
Log in

Use of artificial intelligence in triaging of chest radiographs to reduce radiologists’ workload

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To evaluate whether deep learning–based detection algorithms (DLD)–based triaging can reduce outpatient chest radiograph interpretation workload while maintaining noninferior sensitivity.

Methods

This retrospective study included patients who underwent initial chest radiography at the outpatient clinic between June 1 and June 30, 2017. Readers interpreted radiographs with/without a commercially available DLD that detects nine radiologic findings (atelectasis, calcification, cardiomegaly, consolidation, fibrosis, nodules, pneumothorax, pleural effusion, and pneumoperitoneum). The reading order was determined in a randomized, crossover manner. The radiographs were classified into negative and positive examinations. In a 50% worklist reduction scenario, radiographs were sorted in descending order of probability scores: the lower half was regarded as negative exams, while the remaining were read with DLD by radiologists. The primary analysis evaluated noninferiority in sensitivity between radiologists reading all radiographs and simulating a 50% worklist reduction, with the inferiority margin of 5%. The specificities were compared using McNemar’s test.

Results

The study included 1964 patients (median age [interquartile range], 55 years [40–67 years]). The sensitivity was 82.6% (195 of 236; 95% CI: 77.5%, 87.3%) when readers interpreted all chest radiographs without DLD and 83.5% (197 of 236; 95% CI: 78.8%, 88.1%) in the 50% worklist reduction scenario. The difference in sensitivity was 0.8% (95% CI: − 3.8%, 5.5%), establishing noninferiority of 50% worklist reduction (p = 0.01). The specificity increased from 86.7% (1498 of 1728) to 90.4% (1562 of 1728) (p < 0.001) with DLD-based triage.

Conclusion

Deep learning–based triaging may substantially reduce workload without lowering sensitivity while improving specificity.

Clinical relevance statement

Substantial workload reduction without lowering sensitivity was feasible using deep learning–based triaging of outpatient chest radiograph; however, the legal responsibility for incorrect diagnoses based on AI-standalone interpretation remains an issue that should be defined before clinical implementation.

Key Points

• A 50% workload reduction simulation using deep learning–based detection algorithm maintained noninferior sensitivity while improving specificity.

• The CT recommendation rate significantly decreased in the disease-negative patients, whereas it slightly increased in the disease-positive group without statistical significance.

• In the exploratory analysis, the noninferiority of sensitivity was maintained until 70% of the workload was reduced; the difference in sensitivity was 0%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

AUAFROC:

Area under the alternative free-response receiver operating characteristic curve

DLD:

Deep learning–based detection

IQR:

Interquartile range

References

  1. de Hoop B, Schaefer-Prokop C, Gietema HA et al (2010) Screening for lung cancer with digital chest radiography: sensitivity and number of secondary work-up CT examinations. Radiology 255:629–637

    Article  PubMed  Google Scholar 

  2. Mehrotra P, Bosemani V, Cox J (2009) Do radiologists still need to report chest x rays? Postgrad Med J 85:339–341

    Article  CAS  PubMed  Google Scholar 

  3. Wu JT, Wong KCL, Gur Y et al (2020) Comparison of chest radiograph interpretations by artificial intelligence algorithm vs radiology residents. JAMA Netw Open 3:e2022779

    Article  PubMed  PubMed Central  Google Scholar 

  4. Seah JCY, Tang CHM, Buchlak QD et al (2021) Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit Health 3:e496–e506

    Article  CAS  PubMed  Google Scholar 

  5. Rajpurkar P, Irvin J, Ball RL et al (2018) Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 15:e1002686

    Article  PubMed  PubMed Central  Google Scholar 

  6. Nam JG, Kim M, Park J et al (2021) Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur Respir J 57(5):2003061

  7. Hwang EJ, Nam JG, Lim WH et al (2019) Deep learning for chest radiograph diagnosis in the emergency department. Radiology 293:573–580

    Article  PubMed  Google Scholar 

  8. Cicero M, Bilbily A, Colak E et al (2017) Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest Radiol 52:281–287

    Article  PubMed  Google Scholar 

  9. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G (2019) Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 291:196–202

    Article  PubMed  Google Scholar 

  10. Dunnmon JA, Yi D, Langlotz CP, Re C, Rubin DL, Lungren MP (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537–544

    Article  PubMed  Google Scholar 

  11. Hwang EJ, Park S, Jin KN et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:e191095

    Article  PubMed  PubMed Central  Google Scholar 

  12. Sung J, Park S, Lee SM et al (2021) Added value of deep learning-based detection system for multiple major findings on chest radiographs: a randomized crossover study. Radiology 299:450–459

    Article  PubMed  Google Scholar 

  13. Majkowska A, Mittal S, Steiner DF et al (2020) Chest radiograph interpretation with deep learning models: assessment with radiologist-adjudicated reference standards and population-adjusted evaluation. Radiology 294:421–431

    Article  PubMed  Google Scholar 

  14. Nam JM (1997) Establishing equivalence of two treatments and sample size requirements in matched-pairs design. Biometrics 53:1422–1430

    Article  CAS  PubMed  Google Scholar 

  15. Dorfman DD, Berbaum KS, Metz CE (1992) Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol 27:723–731

    Article  CAS  PubMed  Google Scholar 

  16. Hillis SL (2007) A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med 26:596–619

    Article  MathSciNet  PubMed  PubMed Central  Google Scholar 

  17. The Comprehensive R Archive Network (2022) Package ‘RJafroc’. Available via https://cran.r-project.org/web/packages/RJafroc/RJafroc.pdf. Accessd 26 Feb

  18. Dyer T, Dillard L, Harrison M et al (2021) Diagnosis of normal chest radiographs using an autonomous deep-learning algorithm. Clin Radiol 76:473.e479-473.e415

    Article  Google Scholar 

  19. Kim JH, Kim JY, Kim GH et al (2020) Clinical validation of a deep learning algorithm for detection of pneumonia on chest radiographs in emergency department patients with acute febrile respiratory illness. J Clin Med 9(6):1981

  20. Jang S, Song H, Shin YJ et al (2020) Deep learning-based automatic detection algorithm for reducing overlooked lung cancers on chest radiographs. Radiology 296:652–661

    Article  PubMed  Google Scholar 

Download references

Funding

This study was supported by DongKook Life Science. Co., Ltd., Republic of Korea (grant no. 06–2020-0315); a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant no. HI22C0471); and the Seoul National University Bundang Hospital research fund (grant no. 14–2022-0034).

Author information

Authors and Affiliations

  1. Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea

    Sung Hyun Yoon, Sunyoung Park, Sowon Jang, Junghoon Kim, Kyung Won Lee & Kyung Hee Lee

  2. Department of Public Health Science, Graduate School of Public Health, Seoul National University, Seoul, Korea

    Woojoo Lee

  3. Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea

    Seungjae Lee

  4. Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, PA, USA

    Gabin Yun

  5. Department of Radiology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongnam, Korea

    Kyung Hee Lee

Authors
  1. Sung Hyun Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sunyoung Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sowon Jang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junghoon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kyung Won Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Woojoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Seungjae Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gabin Yun
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Kyung Hee Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kyung Hee Lee.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Kyung Hee Lee.

Conflict of interest

Dr. Kyung Hee Lee received grants from DongKook Life Science. Co., Ltd., Republic of Korea (grant no. 06–2020-0315); Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant no. HI22C0471); and Seoul National University Bundang Hospital research fund (grant no. 14–2022-0034).

Statistics and biometry

Two of the authors have significant statistical expertise.

Informed consent

Written informed consent was waived by the institutional review board.

Ethical approval

Institutional review board approval was obtained.

Study subjects or cohorts overlap

There is no overlap in study subjects or cohorts.

Methodology

• retrospective

• diagnostic study/observational

• performed at one institution

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 308 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yoon, S.H., Park, S., Jang, S. et al. Use of artificial intelligence in triaging of chest radiographs to reduce radiologists’ workload. Eur Radiol 34, 1094–1103 (2024). https://doi.org/10.1007/s00330-023-10124-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-023-10124-1

Keywords

Search

Navigation