Skip to main content

Advertisement

SpringerLink
Log in

COVID-19 Severity Prediction from Chest X-ray Images Using an Anatomy-Aware Deep Learning Model

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

The COVID-19 pandemic has been adversely affecting the patient management systems in hospitals around the world. Radiological imaging, especially chest x-ray and lung Computed Tomography (CT) scans, plays a vital role in the severity analysis of hospitalized COVID-19 patients. However, with an increasing number of patients and a lack of skilled radiologists, automated assessment of COVID-19 severity using medical image analysis has become increasingly important. Chest x-ray (CXR) imaging plays a significant role in assessing the severity of pneumonia, especially in low-resource hospitals, and is the most frequently used diagnostic imaging in the world. Previous methods that automatically predict the severity of COVID-19 pneumonia mainly focus on feature pooling from pre-trained CXR models without explicitly considering the underlying human anatomical attributes. This paper proposes an anatomy-aware (AA) deep learning model that learns the generic features from x-ray images considering the underlying anatomical information. Utilizing a pre-trained model and lung segmentation masks, the model generates a feature vector including disease-level features and lung involvement scores. We have used four different open-source datasets, along with an in-house annotated test set for training and evaluation of the proposed method. The proposed method improves the geographical extent score by 11% in terms of mean squared error (MSE) while preserving the benchmark result in lung opacity score. The results demonstrate the effectiveness of the proposed AA model in COVID-19 severity prediction from chest X-ray images. The algorithm can be used in low-resource setting hospitals for COVID-19 severity prediction, especially where there is a lack of skilled radiologists.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. W.H.O. (2023) WHO Coronavirus (COVID-19) Dashboard With Vaccination Data, https://covid19.who.int/, (Accessed on 03/16/2021)

  2. Irkham I, A. U. Ibrahim, C. W. Nwekwo, F. Al-Turjman, Y. W. Hartati (2022) Current technologies for detection of COVID-19: Biosensors, artificial intelligence and internet of medical things (IOMT). Sensors 23(1):426.

    Article  PubMed  PubMed Central  Google Scholar 

  3. M. Francone, F. Iafrate, G. M. Masci, S. Coco, F. Cilia, L. Manganaro, V. Panebianco, C. Andreoli, M. C. Colaiacomo, M. A. Zingaropoli, et al., Chest ct score in COVID-19 patients: correlation with disease severity and short-term prognosis, European Radiology 30 (12) (2020) 6808–6817.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. S. Punitha, F. Al-Turjman, T. Stephan, Genetically optimized computer-aided diagnosis for detection and classification of COVID-19, AI-powered IoT for COVID-19 (2020) 105–122.

  5. J. T. Wu, K. C. Wong, Y. Gur, N. Ansari, A. Karargyris, A. Sharma, M. Morris, B. Saboury, H. Ahmad, O. Boyko, et al., Comparison of chest radiograph interpretations by artificial intelligence algorithm vs radiology residents, JAMA network open 3 (10) (2020) e2022779–e2022779.

    Article  PubMed  PubMed Central  Google Scholar 

  6. E. Hussain, M. Hasan, M. A. Rahman, I. Lee, T. Tamanna, M. Z. Parvez, Corodet: A deep learning based classification for COVID-19 detection using chest x-ray images, Chaos, Solitons & Fractals 142 (2021) 110495.

    Article  Google Scholar 

  7. R. A. Al-Falluji, Z. D. Katheeth, B. Alathari, Automatic detection of COVID-19 using chest x-ray images and modified resnet18-based convolution neural networks, Comput. Mater. Contin. (2021) 1301–1313.

  8. A. Abbas, M. M. Abdelsamea, M. M. Gaber, Classification of COVID-19 in chest x-ray images using detrac deep convolutional neural network, Appl. Intell. 51 (2) (2021) 854–864.

    Article  Google Scholar 

  9. A. Afifi, N. E. Hafsa, M. A. Ali, A. Alhumam, S. Alsalman, An ensemble of global and local-attention based convolutional neural networks for COVID-19 diagnosis on chest x-ray images, Symmetry 13 (1) (2021) 113.

    Article  CAS  Google Scholar 

  10. M. Umer, I. Ashraf, S. Ullah, A. Mehmood, G. S. Choi, Covinet: a convolutional neural network approach for predicting COVID-19 from chest x-ray images, J. Ambient Intell. Humaniz. Comput. (2021) 1–13.

  11. G. Wang, X. Liu, J. Shen, C. Wang, Z. Li, L. Ye, X. Wu, T. Chen, K. Wang, X. Zhang, et al., A deep-learning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest x-ray images, Nature Biomedical Engineering (2021) 1–13.

  12. X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, R. M. Summers, Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases, in: Proc. IEEE CVPR, 2017, pp. 2097–2106.

  13. J. P. Cohen, L. Dao, K. Roth, P. Morrison, Y. Bengio, A. F. Abbasi, B. Shen, H. K. Mahsa, M. Ghassemi, H. Li, et al., Predicting COVID-19 pneumonia severity on chest x-ray with deep learning, Cureus 12 (7) (2020).

  14. H. J. Koo, S. Lim, J. Choe, S.-H. Choi, H. Sung, K.-H. Do, Radiographic and ct features of viral pneumonia, Radiographics 38 (3) (2018) 719–739.

    Article  PubMed  Google Scholar 

  15. A. Narin, C. Kaya, Z. Pamuk, Automatic detection of coronavirus disease (COVID-19) using x-ray images and deep convolutional neural networks, arXiv preprint arXiv:2003.10849 (2020).

  16. A. M. Ismael, A. Şengür, Deep learning approaches for COVID-19 detection based on chest x-ray images, Expert Systems with Applications 164 (2021) 114054.

    Article  PubMed  Google Scholar 

  17. A. U. Ibrahim, M. Ozsoz, S. Serte, F. Al-Turjman, P. S. Yakoi, Pneumonia classification using deep learning from chest x-ray images during COVID-19, Cognitive Computation (2021) 1–13.

  18. A. S. Al-Waisy, S. Al-Fahdawi, M. A. Mohammed, K. H. Abdulkareem, S. A. Mostafa, M. S. Maashi, M. Arif, B. Garcia-Zapirain, COVID-chexnet: hybrid deep learning framework for identifying COVID-19 virus in chest x-rays images, Soft computing 27 (5) (2023) 2657–2672.

    Article  PubMed  Google Scholar 

  19. A. Hamza, M. A. Khan, M. Alhaisoni, A. Al Hejaili, K. A. Shaban, S. Alsubai, A. Alasiry, M. Marzougui, D2bof-covidnet: A framework of deep bayesian optimization and fusion-assisted optimal deep features for COVID-19 classification using chest x-ray and MRI scans, Diagnostics 13 (1) (2023) 101.

  20. Y. Kaya, E. Gürsoy, A mobilenet-based CNN model with a novel fine-tuning mechanism for COVID-19 infection detection, Soft Computing (2023) 1–15.

  21. K. Jyoti, S. Sushma, S. Yadav, P. Kumar, R. B. Pachori, S. Mukherjee, Automatic diagnosis of COVID-19 with mca-inspired tqwt-based classification of chest x-ray images, Computers in Biology and Medicine 152 (2023) 106331.

    Article  PubMed  Google Scholar 

  22. M. M. Islam, M. Z. Islam, A. Asraf, M. S. Al-Rakhami, W. Ding, A. H. Sodhro, Diagnosis of COVID-19 from x-rays using combined CNN-RNN architecture with transfer learning, BenchCouncil Transactions on Benchmarks, Standards and Evaluations (2023) 100088.

    Google Scholar 

  23. S. Motamed, P. Rogalla, F. Khalvati, Data augmentation using generative adversarial networks (gans) for gan-based detection of pneumonia and COVID-19 in chest x-ray images, arXiv preprint arXiv:2006.03622 (2021).

  24. T. Zebin, S. Rezvy, Covid-19 detection and disease progression visualization: Deep learning on chest x-rays for classification and coarse localization, Appl. Intell. 51 (2) (2021) 1010–1021.

    Article  Google Scholar 

  25. Z. Ullah, M. Usman, S. Latif, J. Gwak, Densely attention mechanism based network for COVID-19 detection in chest x-rays, Scientific Reports 13 (1) (2023) 261.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  26. T. B. Chandra, K. Verma, B. K. Singh, D. Jain, S. S. Netam, Coronavirus disease (COVID-19) detection in chest x-ray images using majority voting based classifier ensemble, Expert Syst. Appl. 165 (2021) 113909.

    Article  PubMed  Google Scholar 

  27. K. K. Singh, A. Singh, Diagnosis of COVID-19 from chest x-ray images using wavelets-based depthwise convolution network, Big Data Mining and Analytics 4 (2) (2021) 84–93.

    Article  Google Scholar 

  28. H. Malik, A. Naeem, R. A. Naqvi, W.-K. Loh, DMFL_net: A federated learning-based framework for the classification of COVID-19 from multiple chest diseases using x-rays, Sensors 23 (2) (2023) 743.

    Article  PubMed  PubMed Central  Google Scholar 

  29. A. Degerli, M. Ahishali, S. Kiranyaz, M. E. Chowdhury, M. Gabbouj, Reliable COVID-19 detection using chest x-ray images, in: 2021 IEEE International Conference on Image Processing (ICIP), IEEE, 2021, pp. 185–189.

  30. A. Haghanifar, M. M. Majdabadi, Y. Choi, S. Deivalakshmi, S. Ko, COVID-CXNET: Detecting COVID-19 in frontal chest x-ray images using deep learning, Multimedia Tools and Applications 81 (21) (2022) 30615–30645.

    Article  PubMed  PubMed Central  Google Scholar 

  31. T. Ozturk, M. Talo, E. A. Yildirim, U. B. Baloglu, O. Yildirim, U. R. Acharya, Automated detection of COVID-19 cases using deep neural networks with x-ray images, Computers in biology and medicine 121 (2020) 103792.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. F. Khozeimeh, D. Sharifrazi, N. H. Izadi, J. H. Joloudari, A. Shoeibi, R. Alizadehsani, J. M. Gorriz, S. Hussain, Z. A. Sani, H. Moosaei, et al., Combining a convolutional neural network with autoencoders to predict the survival chance of COVID-19 patients, Scientific Reports 11 (1) (2021) 15343.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need. in advances in neural information processing systems, in: Proc. NIPS, 2007, pp. 5998–6008.

  34. K. Li, Z. Wu, K.-C. Peng, J. Ernst, Y. Fu, Tell me where to look: Guided attention inference network, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9215–9223.

  35. J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.

  36. N. Sarafianos, X. Xu, I. A. Kakadiaris, Deep imbalanced attribute classification using visual attention aggregation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 680–697.

  37. M. Jaderberg, K. Simonyan, A. Zisserman, et al., Spatial transformer networks, Advances in neural information processing systems 28 (2015).

  38. S. Woo, J. Park, J.-Y. Lee, I. S. Kweon, Cbam: Convolutional block attention module, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.

  39. J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, H. Lu, Dual attention network for scene segmentation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3146–3154.

  40. S. Albahli, N. Ayub, M. Shiraz, Coronavirus disease (COVID-19) detection using x-ray images and enhanced densenet, Applied Soft Computing 110 (2021) 107645.

    Article  PubMed  PubMed Central  Google Scholar 

  41. G. K. Yenurkar, S. Mal, Effective detection of COVID-19 outbreak in chest x-rays using fusionnet model, The Imaging Science Journal (2023) 1–21.

  42. L. T. Duong, P. T. Nguyen, L. Iovino, M. Flammini, Automatic detection of COVID-19 from chest x-ray and lung computed tomography images using deep neural networks and transfer learning, Applied Soft Computing 132 (2023) 109851.

    Article  PubMed  Google Scholar 

  43. H. Mukherjee, S. Ghosh, A. Dhar, S. M. Obaidullah, K. Santosh, K. Roy, Deep neural network to detect COVID-19: one architecture for both CT scans and chest x-rays, Applied Intelligence 51 (2021) 2777–2789.

    Article  PubMed  Google Scholar 

  44. S. Roy, A. K. Das, Deep-cov: An integrated deep learning model to detect COVID-19 using chest x-ray and CT images, Computational Intelligence (2023).

  45. M. Karim, T. Döhmen, D. Rebholz-Schuhmann, S. Decker, M. Cochez, O. Beyan, et al., DeepCOVIDexplainer: Explainable COVID-19 predictions based on chest x-ray images, arXiv preprint arXiv:2004.04582 (2020).

  46. H. Liu, L. Wang, Y. Nan, F. Jin, Q. Wang, J. Pu, Sdfn: Segmentation-based deep fusion network for thoracic disease classification in chest x-ray images, Comput. Med. Imaging Graph. 75 (2019) 66–73.

    Article  PubMed  PubMed Central  Google Scholar 

  47. U. Kamal, M. Zunaed, N. B. Nizam, T. Hasan, Anatomy-xnet: An anatomy aware convolutional neural network for thoracic disease classification in chest x-rays, IEEE Journal of Biomedical and Health Informatics 26 (11) (2022) 5518–5528.

    Article  PubMed  Google Scholar 

  48. T. B. Chandra, K. Verma, D. Jain, S. S. Netam, Localization of the suspected abnormal region in chest radiograph images, in: Proc. IEEE ICPC2T, IEEE, 2020, pp. 204–209.

  49. Ü. Budak, M. Çıbuk, Z. Cömert, A. Şengür, Efficient COVID-19 segmentation from CT slices exploiting semantic segmentation with integrated attention mechanism, Journal of Digital Imaging 34 (2021) 263–272.

    Article  PubMed  PubMed Central  Google Scholar 

  50. J. Zhu, B. Shen, A. Abbasi, M. Hoshmand-Kochi, H. Li, T. Q. Duong, Deep transfer learning artificial intelligence accurately stages COVID-19 lung disease severity on portable chest radiographs, PloS one 15 (7) (2020) e0236621.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Z. Jiao, J. W. Choi, K. Halsey, T. M. L. Tran, B. Hsieh, D. Wang, F. Eweje, R. Wang, K. Chang, J. Wu, et al., Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: a retrospective study, The Lancet Digital Health 3 (5) (2021) e286–e294.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. S. Park, G. Kim, Y. Oh, J. B. Seo, S. M. Lee, J. H. Kim, S. Moon, J.-K. Lim, J. C. Ye, Vision transformer using low-level chest x-ray feature corpus for COVID-19 diagnosis and severity quantification, arXiv preprint arXiv:2104.07235 (2021).

  53. A. Signoroni, M. Savardi, S. Benini, N. Adami, R. Leonardi, P. Gibellini, F. Vaccher, M. Ravanelli, A. Borghesi, R. Maroldi, et al., BS-Net: Learning COVID-19 pneumonia severity on a large chest X-ray dataset, Med. Image Anal. 71 (2021) 102046.

    Article  PubMed  PubMed Central  Google Scholar 

  54. H. Li, P. Xiong, J. An, L. Wang, Pyramid attention network for semantic segmentation. arXiv preprint arXiv:1805.10180 (2018).

  55. W. Ye, J. Yao, H. Xue, Y. Li, Weakly supervised lesion localization with probabilistic-cam pooling, arXiv preprint arXiv:2005.14480 (2020).

  56. A. K. Mondal, A. Agarwal, J. Dolz, C. Desrosiers, Revisiting cyclegan for semi-supervised segmentation, arXiv preprint arXiv:1908.11569 (2019).

  57. J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proc. IEEE CVPR, 2017, pp. 2223–2232.

  58. O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: Proc. MICCAI, Springer, 2015, pp. 234–241.

  59. G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks, in: Proc. IEEE CVPR, 2017, pp. 4700–4708.

  60. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: Proc. IEEE CVPR, 2016, pp. 2921–2929.

  61. J. Shiraishi, S. Katsuragawa, J. Ikezoe, T. Matsumoto, T. Kobayashi, K.-I. Komatsu, M. Matsui, H. Fujita, Y. Kodera, K. Doi, Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules, Am. J. Roentgenol. 174 (1) (2000) 71–74.

    Article  CAS  Google Scholar 

  62. B. Van Ginneken, M. B. Stegmann, M. Loog, Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database, Med. Image Anal. 10 (1) (2006) 19–40.

    Article  PubMed  Google Scholar 

  63. J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al., Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, in: Proc. AAAI Conf. on Artificial Intelligence, Vol. 33, 2019, pp. 590–597.

  64. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: Proc. IEEE CVPR, Ieee, 2009, pp. 248–255.

Download references

Funding

This work was supported by a Special Grant from the Information and Communication Technology (ICT) Division, Bangladesh.

Author information

Authors and Affiliations

  1. mHealth Research Group, Department of Biomedical Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka, 1205, Bangladesh

    Nusrat Binta Nizam, Sadi Mohammad Siddiquee & Taufiq Hasan

  2. Department of Radiology and Imaging, Bangabandhu Sheikh Mujib Medical University, Shahbagh, Dhaka, 1000, Bangladesh

    Mahbuba Shirin

  3. Department of Electrical and Electronics Engineering (EEE), Bangladesh University of Engineering and Technology, Dhaka, 1205, Bangladesh

    Mohammed Imamul Hassan Bhuiyan

  4. Center for Bioengineering Innovation and Design (CBID), Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA

    Taufiq Hasan

Authors
  1. Nusrat Binta Nizam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sadi Mohammad Siddiquee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahbuba Shirin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohammed Imamul Hassan Bhuiyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Taufiq Hasan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Nusrat Binta Nizam: literature review, software/coding, experiments, writing manuscript, editing manuscript, preparing response to review document. Sadi Mohammad Siddiquee: conceptualization, methodology development, software/coding, experiments, writing first draft. Mahbuba Shirin: data collection and annotation. Mohammed Imamul Hassan Bhuiyan: conceptualization, supervision, writing — reviewing and editing. Taufiq Hasan: conceptualization, supervision, writing — reviewing and editing.

Corresponding author

Correspondence to Taufiq Hasan.

Ethics declarations

Ethics Approval

No human subjects were involved in this research. Existing open-access medical image datasets were used for this work.

Consent to Participate

Not applicable, since no human subjects were involved in this research.

Consent for Publication

Not applicable, since no human subjects were involved in this research.

Competing Interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nizam, N.B., Siddiquee, S.M., Shirin, M. et al. COVID-19 Severity Prediction from Chest X-ray Images Using an Anatomy-Aware Deep Learning Model. J Digit Imaging 36, 2100–2112 (2023). https://doi.org/10.1007/s10278-023-00861-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-023-00861-6

Keywords

Search

Navigation