Abstract
The most malignant tumors of the central nervous system are adult-type diffuse gliomas. Historically, glioma subtype classification has been based on morphological features. However, since 2016, WHO recognizes that molecular evaluation is critical for subtyping. Among molecular markers, the mutation status of IDH1 and the codeletion of 1p/19q are crucial for the precise diagnosis of these malignancies. In pathology laboratories, however, manual screening for those markers is time-consuming and susceptible to error. To overcome these limitations, we propose a novel multimodal biomarker classification method that integrates image features derived from brain magnetic resonance imaging and histopathological exams. The proposed model consists of two branches, the first branch takes as input a multi-scale Hematoxylin and Eosin whole slide image, and the second branch uses the pre-segmented region of interest from the magnetic resonance imaging. Both branches are based on convolutional neural networks. After passing the exams by the two embedding branches, the output feature vectors are concatenated, and a multi-layer perceptron is used to classify the glioma biomarkers as a multi-class problem. In this work, several fusion strategies were studied, including a cascade model with mid-fusion; a mid-fusion model, a late fusion model, and a mid-context fusion model. The models were tested using a publicly available data set from The Cancer Genome Atlas. Our cross-validated classification models achieved an area under the curve of 0.874, 0.863, and 0.815 for the proposed multimodal, magnetic resonance imaging, and Hematoxylin and Eosin stain slide images respectively, indicating our multimodal model outperforms its unimodal counterparts and the state-of-the-art glioma biomarker classification methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Albuquerque, T., Cruz, R., Cardoso, J.S.: Ordinal losses for classification of cervical cancer risk. PeerJ Comput. Sci. 7, e457 (2021). https://doi.org/10.7717/peerj-cs.457
Alleman, K., Knecht, E., Huang, J., Zhang, L., Lam, S., DeCuypere, M.: Multimodal deep learning-based prognostication in glioma patients: a systematic review. Cancers 15(2) (2023). https://doi.org/10.3390/cancers15020545. https://www.mdpi.com/2072-6694/15/2/545
Braman, N., Gordon, J.W.H., Goossens, E.T., Willis, C.S., Stumpe, M.C., Venkataraman, J.: Deep orthogonal fusion: multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data. arXiv abs/2107.00648 (2021)
Choi, K.S., Choi, S.H., Jeong, B.: Prediction of IDH genotype in gliomas with dynamic susceptibility contrast perfusion MR imaging using an explainable recurrent neural network. Neuro-Oncology 21(9), 1197–1209 (2019). https://doi.org/10.1093/neuonc/noz095
Choi, Y.S., et al.: Fully automated hybrid approach to predict the IDH mutation status of gliomas via deep learning and radiomics. Neuro-oncology 23(2), 304–313 (2020)
Chunduru, P., Phillips, J.J., Molinaro, A.M.: Prognostic risk stratification of gliomas using deep learning in digital pathology images. Neuro-Oncol. Adv. 4 (2022)
Cui, D., Liu, Y., Liu, G., Liu, L.: A multiple-instance learning-based convolutional neural network model to detect the IDH1 mutation in the histopathology images of glioma tissues. J. Comput. Biol. 27(8), 1264–1272 (2020). https://doi.org/10.1089/cmb.2019.0410
Decuyper, M., Bonte, S., Deblaere, K., Van Holen, R.: Automated MRI based pipeline for segmentation and prediction of grade, IDH mutation and 1p19q co-deletion in glioma. Comput. Med. Imaging Graph. 88, 101831 (2021). https://doi.org/10.1016/j.compmedimag.2020.101831. https://www.sciencedirect.com/science/article/pii/S0895611120301269
Deluche, E., et al.: CHI3L1, NTRK2, 1p/19q and IDH status predicts prognosis in glioma. Cancers 11, 544 (2019)
Gaillard, F.: Diffuse glioma classification (WHO 5th edition, 2021) (2021). https://doi.org/10.53347/rid-94212
Haydar, N., et al.: Role of magnetic resonance imaging (MRI) in grading gliomas comparable with pathology: a cross-sectional study from Syria. Ann. Med. Surg. 82, 104679 (2022). https://doi.org/10.1016/j.amsu.2022.104679. https://www.sciencedirect.com/science/article/pii/S204908012201439X
Jiang, S., Zanazzi, G.J., Hassanpour, S.: Predicting prognosis and IDH mutation status for patients with lower-grade gliomas using whole slide images. Sci. Rep. 11, 16849 (2021)
Kim, D., et al.: Prediction of 1P/19Q codeletion in diffuse glioma patients using pre-operative multiparametric magnetic resonance imaging. Front. Comput. Neurosci. 13 (2019). https://doi.org/10.3389/fncom.2019.00052. https://www.frontiersin.org/articles/10.3389/fncom.2019.00052
Kofler, F., et al.: Brats toolkit: translating brats brain tumor segmentation algorithms into clinical and scientific practice. Front. Neurosci. 14 (2020). https://doi.org/10.3389/fnins.2020.00125. https://www.frontiersin.org/articles/10.3389/fnins.2020.00125
Lee, M.K., Park, J.E., Jo, Y.S., Park, S.Y., Kim, S.J., Kim, H.S.: Advanced imaging parameters improve the prediction of diffuse lower-grade gliomas subtype, IDH mutant with no 1p19q codeletion: added value to the T2/flair mismatch sign. Eur. Radiol. 30, 844–854 (2019)
Li, Y., et al.: Radiomics-based method for predicting the glioma subtype as defined by tumor grade, IDH mutation, and 1p/19q codeletion. Cancers 14, 1778 (2022)
Liechty, B., et al.: Machine learning can aid in prediction of IDH mutation from H &E-stained histology slides in infiltrating gliomas. Sci. Rep. 12, 22623 (2022)
Liu, S., et al.: Isocitrate dehydrogenase (IDH) status prediction in histopathology images of gliomas using deep learning. Sci. Rep. 10, 7733 (2020). https://doi.org/10.1038/s41598-020-64588-y
Louis, D., et al.: The 2021 who classification of tumors of the central nervous system: a summary. Neuro-Oncol. 23, 1231–1251 (2021). https://doi.org/10.1093/neuonc/noab106
Ostrom, Q.T., et al.: CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2012–2016. Neuro-Oncol. 21(Suppl._5), v1–v100 (2019). https://doi.org/10.1093/neuonc/noz150
Rathore, S., Niazi, T., Iftikhar, A., Chaddad, A.: Glioma grading via analysis of digital pathology images using machine learning. Cancers 12, 578 (2020). https://doi.org/10.3390/cancers12030578
Whitfield, B.T., Huse, J.T.: Classification of adult-type diffuse gliomas: impact of the world health organization 2021 update. Brain Pathol. 32, e13062 (2022)
Xue, Y., et al.: Brain tumor classification with tumor segmentations and a dual path residual convolutional neural network from MRI and pathology images. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11993, pp. 360–367. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46643-5_36
Yan, J., et al.: Predicting 1p/19q co-deletion status from magnetic resonance imaging using deep learning in adult-type diffuse lower-grade gliomas: a discovery and validation study. Lab. Invest. 102, 154–159 (2022). https://doi.org/10.1038/s41374-021-00692-5
Yang, X., Lin, Y., Xing, Z., She, D., Su, Y., Cao, D.: Predicting 1p/19q codeletion status using diffusion-, susceptibility-, perfusion-weighted, and conventional MRI in IDH-mutant lower-grade gliomas. Acta Radiol. 62, 1657–1665 (2020)
Zhou, H., et al.: Machine learning reveals multimodal MRI patterns predictive of isocitrate dehydrogenase and 1p/19q status in diffuse low- and high-grade gliomas. J. Neurooncol. 142(2), 299–307 (2019). https://doi.org/10.1007/s11060-019-03096-0
Acknowledgments
The results shown here are in whole based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga. This work is co-financed by Component 5 - Capitalization and Business Innovation of core funding for Technology and Innovation Centres (CTI), integrated in the Resilience Dimension of the Recovery and Resilience Plan within the scope of the Recovery and Resilience Mechanism (MRR) of the European Union (EU), framed in the Next Generation EU, for the period 2021–2026. Tomé Albuquerque was supported by Ph.D. grant 2021.05102.BD provided by FCT.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Albuquerque, T. et al. (2023). Multimodal Context-Aware Detection of Glioma Biomarkers Using MRI and WSI. In: Woo, J., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops. MICCAI 2023. Lecture Notes in Computer Science, vol 14394. Springer, Cham. https://doi.org/10.1007/978-3-031-47425-5_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-47425-5_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47424-8
Online ISBN: 978-3-031-47425-5
eBook Packages: Computer ScienceComputer Science (R0)
