Deep Learning Classification of Cardiomegaly Using Combined Imaging and Non-imaging ICU Data

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12722)


In this paper, we investigate the classification of cardiomegaly using multimodal data, combining imaging data from chest radiography with routinely collected Intensive Care Unit (ICU) data comprising vital sign values, laboratory measurements, and admission metadata. In practice a clinician would assess for the presence of cardiomegaly using a synthesis of multiple sources of data, however, prior machine learning approaches to this task have focused on chest radiographs only. We show that non-imaging ICU data can be used for cardiomegaly classification and propose a novel multimodal network trained simultaneously on both chest radiographs and ICU data. We compare the predictive power of both single-mode approaches with the joint network. We use a subset of data from the publicly available MIMIC-CXR and MIMIC-IV datasets, which contain both chest radiographs and non-imaging ICU data for the same patients. The approach from non-imaging ICU data alone achieves an AUC of 0.684 and the standard chest radiography approach an AUC of 0.840. Our joint model achieves an AUC of 0.880. We conclude that non-imaging ICU data have predictive value for cardiomegaly, and that combining chest radiographs with non-imaging ICU data has the potential to improve model performance for the same subset of patients, with further work required to demonstrate a significant improvement.


Deep learning Chest X-ray Cardiomegaly Multimodal approach 



GP is supported by an NIHR fellowship. AM and LT are supported by the NIHR Oxford Biomedical Research Centre (BRC). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. BWP acknowledges Rutherford Fund at Health Data Research UK (HDR UK) and Nuffield Department of Population Health (NDPH) Senior Research Fellowship.


  1. 1.
    Baltruschat, I.M., Nickisch, H., Grass, M., Knopp, T., Saalbach, A.: Comparison of deep learning approaches for multi-label chest X-ray classification. Sci. Rep. 9(1), 1–10 (2019). Scholar
  2. 2.
    Bressem, K.K., Adams, L.C., Erxleben, C., Hamm, B., Niehues, S.M., Vahldiek, J.L.: Comparing different deep learning architectures for classification of chest radiographs. Sci. Rep. 10(1), 13590 (2020). Scholar
  3. 3.
    Chang, W., et al.: A machine-learning-based prediction method for hypertension outcomes based on medical data. Diagnostics 9(4), 178 (2019). Scholar
  4. 4.
    Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USAGoogle Scholar
  5. 5.
    Davies, M.J.: The cardiomyopathies: an overview. Heart 83(4), 469 (2000). Scholar
  6. 6.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Institute of Electrical and Electronics Engineers, pp. 248–255. (IEEE) (2010).
  7. 7.
    Gessert, N., Nielsen, M., Shaikh, M., Werner, R., Schlaefer, A.: Skin lesion classification using ensembles of multi-resolution Efficient Nets with metadata. MethodsX 7, 100864 (2020). Scholar
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 2016, pp. 770–778. IEEE Computer Society (2016).
  9. 9.
    Irvin, J., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019. vol. 33, pp. 590–597. AAAI Press (2019).,
  10. 10.
    Ishida, T., Katsuragawa, S., Chida, K., MacMahon, H., Doi, K.: Computer-aided diagnosis for detection of cardiomegaly in digital chest radiographs. In: Medical Imaging 2005: Image Processing. vol. 5747, p. 914. SPIE (2005).
  11. 11.
    Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L.A., Mark, R.: MIMIC-IV v0.4. Technical Report (2020).
  12. 12.
    Johnson, A.E.W., et al.: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data 6(1), 1–8 (2019). Scholar
  13. 13.
    Johnson, A.E.W., et al.: MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv (2019)Google Scholar
  14. 14.
    Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings (2015)Google Scholar
  15. 15.
    Peng, Y., Wang, X., Lu, L., Bagheri, M., Summers, R., Lu, Z.: NegBio: a high-performance tool for negation and uncertainty detection in radiology reports (2017)Google Scholar
  16. 16.
    Pimentel, M., Mahdi, A., Redfern, O., Santos, M., Tarassenko, L.: Uncertainty-aware model for reliable prediction of sepsis in the ICU. In: 2019 Computing in Cardiology Conference (CinC). vol. 45 (2019).
  17. 17.
    Pimentel, M.A.F., et al.: Detecting deteriorating patients in hospital: development and validation of a novel scoring system. Am. J. Respir. Crit. Care Med. (2021). Scholar
  18. 18.
    Que, Q., et al.: CardioXNet: automated detection for cardiomegaly based on deep learning. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS. vol. 2018, pp. 612–615. Institute of Electrical and Electronics Engineers Inc. (2018).
  19. 19.
    Rajpurkar, P., et al.: CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv (2017).
  20. 20.
    Samak, Z.A., Clatworthy, P., Mirmehdi, M.: Prediction of thrombectomy functional outcomes using multimodal data. In: Papież, B.W., Namburete, A.I.L., Yaqub, M., Noble, J.A. (eds.) MIUA 2020. CCIS, vol. 1248, pp. 267–279. Springer, Cham (2020). Scholar
  21. 21.
    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)Google Scholar
  22. 22.
    Smith, L.N.: Cyclical learning rates for training neural networks. In: Proceedings - 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017, pp. 464–472. Institute of Electrical and Electronics Engineers Inc. (2017).
  23. 23.
    Sogancioglu, E., Murphy, K., Calli, E., Scholten, E.T., Schalekamp, S., Van Ginneken, B.: Cardiomegaly detection on chest radiographs: segmentation versus classification. IEEE Access 8, 94631–94642 (2020). Scholar
  24. 24.
    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. vol. 2017, pp. 3462–3471. Institute of Electrical and Electronics Engineers Inc. (2017).

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.Department of Engineering ScienceUniversity of OxfordOxfordUK
  2. 2.Big Data Institute, Li Ka Shing Centre for Health Information and DiscoveryUniversity of OxfordOxfordUK
  3. 3.Kadoorie Centre and Intensive Care Registrar, Thames Valley DeaneryNIHR Academic Clinical Fellow at Oxford UniversityOxfordUK

Personalised recommendations