Abstract
The progression and treatment response of cancer largely depends on the complex tissue structure that surrounds cancer cells in a tumour, known as the tumour microenvironment (TME). Recent technical advances have led to the development of highly multiplexed imaging techniques such as Imaging Mass Cytometry (IMC), which capture the complexity of the TME by producing spatial tissue maps of dozens of proteins. Combining these multidimensional cell phenotypes with their spatial organization to predict clinically relevant information is a challenging computational task and so far no method has addressed it directly. Here, we propose and evaluate MULTIPLAI, a novel framework to predict clinical biomarkers from IMC data. The method relies on attention-based graph neural networks (GNNs) that integrate both the phenotypic and spatial dimensions of IMC images. In this proof-of-concept study we used MULTIPLAI to predict oestrogen receptor (ER) status, a key clinical variable for breast cancer patients. We trained different architectures of our framework on 240 samples and benchmarked against graph learning via graph kernels. Propagation Attribute graph kernels achieved a class-balanced accuracy of 66.18% in the development set (N = 104) while GNNs achieved a class-balanced accuracy of 90.00% on the same set when using the best combination of graph convolution and pooling layers. We further validated this architecture in internal (N = 112) and external test sets from different institutions (N = 281 and N = 350), demonstrating the generalizability of the method. Our results suggest that MULTIPLAI captures important TME features with clinical importance. This is the first application of GNNs to this type of data and opens up new opportunities for predictive modelling of highly multiplexed images.
Keywords
- Graph neural networks
- Highly multiplexed imaging
- Imaging mass cytometry
- Tumour microenvironment
- Breast cancer
M. Crispin-Ortuzar and F. Markowetz—Shared senior authorship.
This is a preview of subscription content, access via your institution.
Buying options
References
Ali, H.R., et al.: Imaging mass cytometry and multiplatform genomics define the phenogenomic landscape of breast cancer. Nat. Cancer 1(2), 163–175 (2020). https://doi.org/10.1038/s43018-020-0026-6
Borgwardt, K.M., Ghisu, M.E., Llinares-López, F., O’Bray, L., Rieck, B.: Graph kernels: state-of-the-art and future challenges. CoRR abs/2011.0 (2020). https://arxiv.org/abs/2011.03854
Bronstein, M.M., Bruna, J., Lecun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond Euclidean data, July 2017. https://doi.org/10.1109/MSP.2017.2693418
Cao, W., Yan, Z., He, Z., He, Z.: A comprehensive survey on geometric deep learning. IEEE Access 8, 35929–35949 (2020). https://doi.org/10.1109/ACCESS.2020.2975067
Chang, Q., Ornatsky, O.I., Siddiqui, I., Loboda, A., Baranov, V.I., Hedley, D.W.: Imaging mass cytometry. Cytometry Part A 91(2), 160–169 (2017). https://doi.org/10.1002/cyto.a.23053. http://doi.wiley.com/10.1002/cyto.a.23053
Curtis, C., et al.: The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature 486(7403), 346–352 (2012). https://doi.org/10.1038/nature10983. https://www.nature.com/articles/nature10983
Giesen, C., et al.: Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat. Methods 11(4), 417–422 (2014). https://doi.org/10.1038/nmeth.2869. https://www.nature.com/articles/nmeth.2869
Hao, S.J., Wan, Y., Xia, Y.Q., Zou, X., Zheng, S.Y.: Size-based separation methods of circulating tumor cells, February 2018. https://doi.org/10.1016/j.addr.2018.01.002
Jackson, H.W., et al.: The single-cell pathology landscape of breast cancer. Nature 578(7796), 615–620 (2020). https://doi.org/10.1038/s41586-019-1876-x
Jaume, G., et al.: Quantifying explainers of graph neural networks in computational pathology. Technical report (2021)
Jaume, G., et al.: Towards explainable graph representations in digital pathology. Technical report (2020)
Kamińska, K., et al.: The role of the cell-cell interactions in cancer progression. J. Cell. Mol. Medi. 19(2), 283–296 (2015). https://doi.org/10.1111/jcmm.12408
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, September 2016. http://arxiv.org/abs/1609.02907
Kong, K., et al.: FLAG: adversarial data augmentation for graph neural networks, October 2020. http://arxiv.org/abs/2010.09891
Kriege, N.M., Johansson, F.D., Morris, C.: A survey on graph kernels. Appl. Netw. Sci. 5(1), 6 (2020). https://doi.org/10.1007/s41109-019-0195-3
Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. In: 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, November 2015. http://arxiv.org/abs/1511.05493
Neumann, M., Garnett, R., Bauckhage, C., Kersting, K.: Propagation kernels: efficient graph kernels from propagated information. Mach. Learn. 102(2), 209–245 (2016). https://doi.org/10.1007/s10994-015-5517-9
Onitilo, A.A., Engel, J.M., Greenlee, R.T., Mukesh, B.N.: Breast cancer subtypes based on ER/PR and Her2 expression: comparison of clinicopathologic features and survival. Clin. Med. Res. 7(1–2), 4–13 (2009). https://doi.org/10.3121/cmr.2008.825. http://www.clinmedres.org/content/7/1-2/4.full
Putti, T.C., et al.: Estrogen receptor-negative breast carcinomas: a review of morphology and immunophenotypical analysis. Mod. Pathol. 18(1), 26–35 (2005). https://doi.org/10.1038/modpathol.3800255
Schapiro, D., et al.: histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. Nat. Methods 14(9), 873–876 (2017). https://doi.org/10.1038/nmeth.4391. http://www.nature.com/articles/nmeth.4391
Somarakis, A., Van Unen, V., Koning, F., Lelieveldt, B., Hollt, T.: ImaCytE: visual exploration of cellular micro-environments for imaging mass cytometry data. IEEE Trans. Vis. Comput. Graph. 27(1), 98–110 (2021). https://doi.org/10.1109/TVCG.2019.2931299
Turajlic, S., Swanton, C.: Implications of cancer evolution for drug development, July 2017. https://doi.org/10.1038/nrd.2017.78. http://tracerx.co.uk/
Wang, F., Tax, D.M.J.: Survey on the attention based RNN model and its applications in computer vision, January 2016. http://arxiv.org/abs/1601.06823
Whiteside, T.L.: The tumor microenvironment and its role in promoting tumor growth, October 2008. https://doi.org/10.1038/onc.2008.271. /pmc/articles/PMC3689267/
Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv, October 2018. http://arxiv.org/abs/1810.00826
Zhao, T., Liu, Y., Neves, L., Woodford, O., Jiang, M., Shah, N.: Data augmentation for graph neural networks. arXiv, July 2020. http://arxiv.org/abs/2006.06830
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Martin-Gonzalez, P., Crispin-Ortuzar, M., Markowetz, F. (2021). Predictive Modelling of Highly Multiplexed Tumour Tissue Images by Graph Neural Networks. In: , et al. Interpretability of Machine Intelligence in Medical Image Computing, and Topological Data Analysis and Its Applications for Medical Data. IMIMIC TDA4MedicalData 2021 2021. Lecture Notes in Computer Science(), vol 12929. Springer, Cham. https://doi.org/10.1007/978-3-030-87444-5_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-87444-5_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87443-8
Online ISBN: 978-3-030-87444-5
eBook Packages: Computer ScienceComputer Science (R0)
-
Published in cooperation with
http://miccai.org/
