Abstract
Previous deep learning efforts have focused on improving the performance of Pulmonary Embolism (PE) diagnosis from Computed Tomography (CT) scans using Convolutional Neural Networks (CNN). However, the features from CT scans alone are not always sufficient for the diagnosis of PE. CT scans along with electronic heath records (EHR) can provide a better insight into the patient’s condition and can lead to more accurate PE diagnosis. In this paper, we propose Pulmonary Embolism Detection using Contrastive Learning (PECon), a supervised contrastive pretraining strategy that employs both the patient’s CT scans as well as the EHR data, aiming to enhance the alignment of feature representations between the two modalities and leverage information to improve the PE diagnosis. In order to achieve this, we make use of the class labels and pull the sample features of the same class together, while pushing away those of the other class. Results show that the proposed work outperforms the existing techniques and achieves state-of-the-art performance on the RadFusion dataset with an F1-score of 0.913, accuracy of 0.90 and an AUROC of 0.943. Furthermore, we also explore the explainability of our approach in comparison to other methods. Our code is publicly available at https://github.com/BioMedIA-MBZUAI/PECon.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bělohlávek, J., Dytrych, V., Linhart, A.: Pulmonary embolism, part I: epidemiology, risk factors and risk stratification, pathophysiology, clinical presentation, diagnosis and nonthrombotic pulmonary embolism. Exp. Clin. Cardiol. 18(2), 129–138 (2013)
Tarbox, A., Swaroop, M.: Pulmonary embolism. Int. J. Crit. Illn. Inj. Sci. 3(1), 69–72 (2013). https://doi.org/10.4103/2229-5151.109427, https://www.ijciis.org/text.asp?2013/3/1/69/109427
Hendriksen, J.M.T., et al.: Clinical characteristics associated with diagnostic delay of pulmonary embolism in primary care: a retrospective observational study. BMJ Open 7(3), e012789 (2017). https://doi.org/10.1136/bmjopen-2016-012789
Alonso-Martínez, J.L., Sánchez, F.J., Echezarreta, M.A.: Delay and misdiagnosis in sub-massive and non-massive acute pulmonary embolism. Eur. J. Intern. Med. 21, 278–282 (2010). https://doi.org/10.1016/J.EJIM.2010.04.005, https://pubmed.ncbi.nlm.nih.gov/20603035/
Masutani, Y., MacMahon, H., Doi, K.: Computerized detection of pulmonary embolism in spiral CT angiography based on volumetric image analysis. IEEE Trans. Med. Imaging 21(12), 1517–1523 (2002). https://doi.org/10.1109/TMI.2002.806586
Liang, J., Bi, J.: Computer aided detection of pulmonary embolism with tobogganing and mutiple instance classification in CT pulmonary angiography. In: Karssemeijer, N., Lelieveldt, B. (eds.) IPMI 2007. LNCS, vol. 4584, pp. 630–641. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73273-0_52
Lin, Y., et al.: Automated pulmonary embolism detection from CTPA images using an end-to-end convolutional neural network. In: Shen, D. (ed.) MICCAI 2019. LNCS, vol. 11767, pp. 280–288. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_31
Khachnaoui, H., Agrébi, M., Halouani, S., Khlifa, N.: Deep learning for automatic pulmonary embolism identification using CTA images. In: 2022 6th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pp. 1–6 (2022). https://doi.org/10.1109/ATSIP55956.2022.9805929
Suman, S., et al.: Attention based CNN-LSTM network for pulmonary embolism prediction on chest computed tomography pulmonary angiograms (2021). https://arxiv.org/abs/2107.06276
Huang, S.C., et al.: PENet-a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging. NPJ Digital Med. 3(1), 61 (2020). https://doi.org/10.1038/s41746-020-0266-y
Li, H., Fan, Y.: Early prediction of Alzheimer’s disease dementia based on baseline hippocampal MRI and 1-year follow-up cognitive measures using deep recurrent neural networks. In: Proceedings. IEEE International Symposium on Biomedical Imaging 2019, pp. 368–371 (2019). https://doi.org/10.1109/ISBI.2019.8759397
Kawahara, J., Daneshvar, S., Argenziano, G., Hamarneh, G.: Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE J. Biomed. Health Inform. 23(2), 538–546 (2019). https://doi.org/10.1109/JBHI.2018.2824327
Huang, S.C., Pareek, A., Zamanian, R., Banerjee, I., Lungren, M.P.: Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Sci. Rep. 10(1), 1–9 (2020)
Zhou, Y., et al.: RadFusion: benchmarking performance and fairness for multimodal pulmonary embolism detection from CT and EHR (2021). https://arxiv.org/abs/2111.11665
Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference, pp. 2–25. PMLR (2022)
Zhang, S., et al.: Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 (2023)
Gu, Y., et al.: Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. (HEALTH) 3(1), 1–23 (2021)
Khosla, P., et al.: Supervised contrastive learning. Adv. Neural. Inf. Process. Syst. 33, 18661–18673 (2020)
Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., Zisserman, A.: A short note about kinetics-600 (2018). arXiv:1808.01340
Albrecht, M.H., et al.: State-of-the-art pulmonary CT angiography for acute pulmonary embolism. Am. J. Roentgenol. 208(3), 495–504 (2016). https://doi.org/10.2214/AJR.16.17202
Oord, A.V.D., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow Twins: self-supervised learning via redundancy reduction (2021). https://arxiv.org/abs/2103.03230
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sanjeev, S., Al Khatib, S.K., Shaaban, M.A., Almakky, I., Papineni, V.R., Yaqub, M. (2024). PECon: Contrastive Pretraining to Enhance Feature Alignment Between CT and EHR Data for Improved Pulmonary Embolism Diagnosis. In: Cao, X., Xu, X., Rekik, I., Cui, Z., Ouyang, X. (eds) Machine Learning in Medical Imaging. MLMI 2023. Lecture Notes in Computer Science, vol 14348. Springer, Cham. https://doi.org/10.1007/978-3-031-45673-2_43
Download citation
DOI: https://doi.org/10.1007/978-3-031-45673-2_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45672-5
Online ISBN: 978-3-031-45673-2
eBook Packages: Computer ScienceComputer Science (R0)
