Abstract
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables into a comprehensive multimodal risk score. Prognostic embeddings from each modality are learned and combined via attention-gated tensor fusion. To maximize the information gleaned from each modality, we introduce a multimodal orthogonalization (MMO) loss term that increases model performance by incentivizing constituent embeddings to be more complementary. DOF predicts OS in glioma patients with a median C-index of 0.788 ± 0.067, significantly outperforming (p = 0.023) the best performing unimodal model with a median C-index of 0.718 ± 0.064. The prognostic model significantly stratifies glioma patients by OS within clinical subsets, adding further granularity to prognostic clinical grading and molecular subtyping.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
El-Deiry, W.S., et al.: The current state of molecular testing in the treatment of patients with solid tumors, 2019. CA Cancer J. Clin. 69(4), 305–343 (2019)
Gillies, R.J., Kinahan, P.E., Hricak, H.: Radiomics: images are more than pictures, they are data. Radiology 278(2), 563–577 (2015)
Saba, L., et al.: The present and future of deep learning in radiology. Eur. J. Radiol. 114, 14–24 (2019)
Skrede, O.-J., et al.: Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. Lancet (London England) 395(10221), 350–360 (2020)
Louis, D.N., et al.: The 2016 world health organization classification of tumors of the central nervous system: a summary. Acta Neuropathologica 131(6), 803–820 (2016)
Siegel, R.L., Miller, K.D., Jemal, A.: Cancer statistics, 2017. CA Cancer J. Clin. 67(1), 7–30 (2017)
Olar, A., Aldape, K.D.: Using the molecular classification of glioblastoma to inform personalized treatment. J. Pathol. 232(2), 165–177 (2014)
Stupp, R., et al.: Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. New Engl. J. Med. 352(10), 987–996 (2005)
Parker, N.R., et al.: Intratumoral heterogeneity identified at the epigenetic, genetic and transcriptional level in glioblastoma. Sci. Rep. 6, 22477 (2016)
Bae, S., et al.: Radiomic MRI phenotyping of glioblastoma: improving survival prediction. Radiology 289(3), 797–806 (2018)
Prasanna, P., et al.: Radiomic features from the peritumoral brain parenchyma on treatment-Naïve multi-parametric MR imaging predict long versus short-term survival in glioblastoma multiforme: preliminary findings. Eur. Radiol. 27(10), 4188–4197 (2017)
Beig, N., et al.: Radiogenomic-based survival risk stratification of tumor habitat on Gd-T1w MRI is associated with biological processes in glioblastoma. Clin. Cancer Res. 26(8), 1866–1876 (2020)
Beig, N., et al.: Sexually dimorphic radiogenomic models identify distinct imaging and biological pathways that are prognostic of overall survival in glioblastoma. Neuro-Oncology 23(2), 251–263 (2021)
Chen, R.J., et al.: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. arXiv:1912.08937 [cs, q-bio]. version: 1, 18 December 2019
Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl. Acad. Sci. 115(13), E2970–E2979 (2018)
Cheerla, A., Gevaert, O.: Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35(14), i446–i454 (2019)
Vaidya, P., et al.: RaPtomics: integrating radiomic and pathomic features for predicting recurrence in early stage lung cancer. In: Medical Imaging 2018: Digital Pathology, vol. 10581, p. 105810M. International Society for Optics and Photonics, 6 March 2018
Subramanian, V., Do, M.N., Syeda-Mahmood, T.: Multimodal fusion of imaging and genomics for lung cancer recurrence prediction. arXiv:2002.01982 [cs, eess, q-bio], 5 February 2020
Zadeh, A., et al.: Tensor fusion network for multimodal sentiment analysis. arXiv:1707.07250 [cs], 23 July 2017
Lezama, J., et al.: O \(\backslash \) Ln’E: orthogonal low-rank embedding, a plug and play geometric loss for deep learning. arXiv:1712.01727 [cs, stat], 5 December 2017
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs], 10 April 2015
Deng, J., et al.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 248–255 (2009). ISSN: 1063–6919
Scarpace, L., et al.: Radiology Data from The Cancer Genome Atlas Glioblas-toma Multiforme [TCGA-GBM] collection. In collab. with TCIA Team. type: dataset (2016)
Pedano, N., et al.: Radiology Data from The Cancer Genome Atlas Low Grade Glioma [TCGA-LGG] collection. In collab. with TCIA Team. type: dataset (2016)
Clark, K., et al.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26(6), 1045–1057 (2013)
Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18(1), 50–60 (1947)
Ching, T., Zhu, X., Garmire, L.X.: Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data. PLOS Comput. Biol. 14(4), e1006076 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Braman, N., Gordon, J.W.H., Goossens, E.T., Willis, C., Stumpe, M.C., Venkataraman, J. (2021). Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12905. Springer, Cham. https://doi.org/10.1007/978-3-030-87240-3_64
Download citation
DOI: https://doi.org/10.1007/978-3-030-87240-3_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87239-7
Online ISBN: 978-3-030-87240-3
eBook Packages: Computer ScienceComputer Science (R0)
-
Published in cooperation with
http://miccai.org/
