Abstract
Chest radiographs (CXRs) are often the primary front-line diagnostic imaging modality. Pulmonary diseases manifest as characteristic changes in lung tissue texture rather than anatomical structure. Hence, we expect that studying changes in only lung tissue texture without the influence of possible structure variations would be advantageous for downstream prognostic and predictive modeling tasks. In this paper, we propose a generative framework, Lung Swapping Autoencoder (LSAE), that learns a factorized representation of a CXR to disentangle the tissue texture representation from the anatomic structure representation. Upon learning the disentanglement, we leverage LSAE in two applications. 1) After adapting the texture encoder in LSAE to a thoracic disease classification task on the large-scale ChestX-ray14 database (N = 112,120), we achieve a competitive result (mAUC: 79.0\(\%\)) with unsupervised pre-training. Moreover, when compared with Inception v3 on our multi-institutional COVID-19 dataset, COVOC (N = 340), for a COVID-19 outcome prediction task (estimating need for ventilation), the texture encoder achieves 13\(\%\) less error with a 77\(\%\) smaller model size, further demonstrating the efficacy of texture representation for lung diseases. 2) We leverage the LSAE for data augmentation by generating hybrid lung images with textures and labels from the COVOC training data and lung structures from ChestX-ray14. This further improves ventilation outcome prediction on COVOC.
The code is available here: https://github.com/cvlab-stonybrook/LSAE.
Research reported in this publication was enabled by the Renaissance School of Medicine at Stony Brook University’s “COVID-19 Data Commons and Analytic Environment”, a data quality initiative instituted by the Office of the Dean, and supported by the Department of Biomedical Informatics. Research was supported by SBU OVPR and IEDM seed grant 2019 (P.P, D.S), NIGMS T32GM008444 (J.B). D.S was partially supported by the Partner University Fund, the SUNY2020 Infrastructure Transportation Security Center, and a gift from Adobe.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bae, J., et al.: Predicting mechanical ventilation requirement and mortality in COVID-19 using radiomics and deep learning on chest radiographs: a multi-institutional study. arXiv preprint arXiv:2007.08028 (2020)
Cohen, J.P., Morrison, P., Dao, L., Roth, K., Duong, T.Q., Ghassemi, M.: COVID-19 image data collection: Prospective predictions are the future. arXiv:2006.11988 (2020)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (2014)
Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297–304. JMLR Workshop and Conference Proceedings (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H.: Patch-based convolutional neural network for whole slide tissue image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2424–2433 (2016)
Hu, Q., Drukker, K., Giger, M.L.: Role of standard and soft tissue chest radiography images in COVID-19 diagnosis using deep learning. In: Medical Imaging 2021: Computer-Aided Diagnosis, vol. 11597, p. 1159704. International Society for Optics and Photonics, February 2021
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
Konwer, A., et al.: Predicting COVID-19 lung infiltrate progression on chest radiographs using spatio-temporal LSTM based encoder-decoder network. In: Medical Imaging with Deep Learning (2021)
Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3911–3919 (2017)
Li, Z., et al.: A novel multiple instance learning framework for COVID-19 severity assessment via data augmentation and self-supervised learning. arXiv:2102.03837 [cs, eess], February 2021
Litmanovich, D.E., Chung, M., Kirkbride, R.R., Kicska, G., Kanne, J.P.: Review of chest radiograph findings of COVID-19 pneumonia and suggested reporting language. J. Thorac. Imaging 35(6), 354–360 (2020)
López-Cabrera, J.D., Orozco-Morales, R., Portal-Diaz, J.A., Lovelle-Enríquez, O., Pérez-Díaz, M.: Current limitations to identify COVID-19 using artificial intelligence with chest X-ray imaging. Heal. Technol. 11(2), 411–424 (2021). https://doi.org/10.1007/s12553-021-00520-2
Park, T., Efros, A.A., Zhang, R., Zhu, J.-Y.: Contrastive learning for unpaired image-to-image translation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 319–345. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_19
Park, T., Zhu, J.Y., Wang, O., Lu, J., Shechtman, E., Efros, A.A., Zhang, R.: Swapping autoencoder for deep image manipulation. arXiv preprint arXiv:2007.00653 (2020)
Rajpurkar, P., et al.: Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
Rott Shaham, T., Dekel, T., Michaeli, T.: Singan: learning a generative model from a single natural image. In: IEEE International Conference on Computer Vision (ICCV) (2019)
Salehi, S., Abedi, A., Balakrishnan, S., Gholamrezanezhad, A.: Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients. Am. J. Roentgenol. 215(1), 87–93 (2020)
Sandfort, V., Yan, K., Pickhardt, P.J., Summers, R.M.: Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Sci. Rep. 9(1), 1–9 (2019)
Shin, H., et al.: Medical image synthesis for data augmentation and anonymization using generative adversarial networks. CoRR abs/1807.10225 (2018)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Toussie, D., et al.: Clinical and chest radiography features determine patient outcomes in young and middle age adults with COVID-19. Radiology 201754 (2020)
Wang, H., Xia, Y.: Chestnet: a deep neural network for classification of thoracic diseases on chest radiography. arXiv preprint arXiv:1807.03058 (2018)
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (2017)
Wong, H.Y.F., et al.: Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 201160 (2020)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Acknowledgement
We thank Amit Gupta, Nicole Sakla, and Rishabh Gattu for their expert evaluation of the generated radiographs.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, L. et al. (2021). Chest Radiograph Disentanglement for COVID-19 Outcome Prediction. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12907. Springer, Cham. https://doi.org/10.1007/978-3-030-87234-2_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-87234-2_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87233-5
Online ISBN: 978-3-030-87234-2
eBook Packages: Computer ScienceComputer Science (R0)
