Skip to main content

Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12905)


Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables into a comprehensive multimodal risk score. Prognostic embeddings from each modality are learned and combined via attention-gated tensor fusion. To maximize the information gleaned from each modality, we introduce a multimodal orthogonalization (MMO) loss term that increases model performance by incentivizing constituent embeddings to be more complementary. DOF predicts OS in glioma patients with a median C-index of 0.788 ± 0.067, significantly outperforming (p = 0.023) the best performing unimodal model with a median C-index of 0.718 ± 0.064. The prognostic model significantly stratifies glioma patients by OS within clinical subsets, adding further granularity to prognostic clinical grading and molecular subtyping.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. El-Deiry, W.S., et al.: The current state of molecular testing in the treatment of patients with solid tumors, 2019. CA Cancer J. Clin. 69(4), 305–343 (2019)

    Google Scholar 

  2. Gillies, R.J., Kinahan, P.E., Hricak, H.: Radiomics: images are more than pictures, they are data. Radiology 278(2), 563–577 (2015)

    CrossRef  Google Scholar 

  3. Saba, L., et al.: The present and future of deep learning in radiology. Eur. J. Radiol. 114, 14–24 (2019)

    CrossRef  Google Scholar 

  4. Skrede, O.-J., et al.: Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. Lancet (London England) 395(10221), 350–360 (2020)

    CrossRef  Google Scholar 

  5. Louis, D.N., et al.: The 2016 world health organization classification of tumors of the central nervous system: a summary. Acta Neuropathologica 131(6), 803–820 (2016)

    CrossRef  Google Scholar 

  6. Siegel, R.L., Miller, K.D., Jemal, A.: Cancer statistics, 2017. CA Cancer J. Clin. 67(1), 7–30 (2017)

    CrossRef  Google Scholar 

  7. Olar, A., Aldape, K.D.: Using the molecular classification of glioblastoma to inform personalized treatment. J. Pathol. 232(2), 165–177 (2014)

    CrossRef  Google Scholar 

  8. Stupp, R., et al.: Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. New Engl. J. Med. 352(10), 987–996 (2005)

    CrossRef  Google Scholar 

  9. Parker, N.R., et al.: Intratumoral heterogeneity identified at the epigenetic, genetic and transcriptional level in glioblastoma. Sci. Rep. 6, 22477 (2016)

    CrossRef  Google Scholar 

  10. Bae, S., et al.: Radiomic MRI phenotyping of glioblastoma: improving survival prediction. Radiology 289(3), 797–806 (2018)

    CrossRef  Google Scholar 

  11. Prasanna, P., et al.: Radiomic features from the peritumoral brain parenchyma on treatment-Naïve multi-parametric MR imaging predict long versus short-term survival in glioblastoma multiforme: preliminary findings. Eur. Radiol. 27(10), 4188–4197 (2017)

    CrossRef  Google Scholar 

  12. Beig, N., et al.: Radiogenomic-based survival risk stratification of tumor habitat on Gd-T1w MRI is associated with biological processes in glioblastoma. Clin. Cancer Res. 26(8), 1866–1876 (2020)

    CrossRef  Google Scholar 

  13. Beig, N., et al.: Sexually dimorphic radiogenomic models identify distinct imaging and biological pathways that are prognostic of overall survival in glioblastoma. Neuro-Oncology 23(2), 251–263 (2021)

    CrossRef  Google Scholar 

  14. Chen, R.J., et al.: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. arXiv:1912.08937 [cs, q-bio]. version: 1, 18 December 2019

  15. Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl. Acad. Sci. 115(13), E2970–E2979 (2018)

    CrossRef  Google Scholar 

  16. Cheerla, A., Gevaert, O.: Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35(14), i446–i454 (2019)

    CrossRef  Google Scholar 

  17. Vaidya, P., et al.: RaPtomics: integrating radiomic and pathomic features for predicting recurrence in early stage lung cancer. In: Medical Imaging 2018: Digital Pathology, vol. 10581, p. 105810M. International Society for Optics and Photonics, 6 March 2018

    Google Scholar 

  18. Subramanian, V., Do, M.N., Syeda-Mahmood, T.: Multimodal fusion of imaging and genomics for lung cancer recurrence prediction. arXiv:2002.01982 [cs, eess, q-bio], 5 February 2020

  19. Zadeh, A., et al.: Tensor fusion network for multimodal sentiment analysis. arXiv:1707.07250 [cs], 23 July 2017

  20. Lezama, J., et al.: O \(\backslash \) Ln’E: orthogonal low-rank embedding, a plug and play geometric loss for deep learning. arXiv:1712.01727 [cs, stat], 5 December 2017

  21. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs], 10 April 2015

  22. Deng, J., et al.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 248–255 (2009). ISSN: 1063–6919

    Google Scholar 

  23. Scarpace, L., et al.: Radiology Data from The Cancer Genome Atlas Glioblas-toma Multiforme [TCGA-GBM] collection. In collab. with TCIA Team. type: dataset (2016)

    Google Scholar 

  24. Pedano, N., et al.: Radiology Data from The Cancer Genome Atlas Low Grade Glioma [TCGA-LGG] collection. In collab. with TCIA Team. type: dataset (2016)

    Google Scholar 

  25. Clark, K., et al.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26(6), 1045–1057 (2013)

    CrossRef  Google Scholar 

  26. Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18(1), 50–60 (1947)

    CrossRef  MathSciNet  Google Scholar 

  27. Ching, T., Zhu, X., Garmire, L.X.: Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data. PLOS Comput. Biol. 14(4), e1006076 (2018)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Nathaniel Braman .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7115 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Braman, N., Gordon, J.W.H., Goossens, E.T., Willis, C., Stumpe, M.C., Venkataraman, J. (2021). Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12905. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87239-7

  • Online ISBN: 978-3-030-87240-3

  • eBook Packages: Computer ScienceComputer Science (R0)