Skip to main content

Predictive Modelling of Highly Multiplexed Tumour Tissue Images by Graph Neural Networks

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12929)


The progression and treatment response of cancer largely depends on the complex tissue structure that surrounds cancer cells in a tumour, known as the tumour microenvironment (TME). Recent technical advances have led to the development of highly multiplexed imaging techniques such as Imaging Mass Cytometry (IMC), which capture the complexity of the TME by producing spatial tissue maps of dozens of proteins. Combining these multidimensional cell phenotypes with their spatial organization to predict clinically relevant information is a challenging computational task and so far no method has addressed it directly. Here, we propose and evaluate MULTIPLAI, a novel framework to predict clinical biomarkers from IMC data. The method relies on attention-based graph neural networks (GNNs) that integrate both the phenotypic and spatial dimensions of IMC images. In this proof-of-concept study we used MULTIPLAI to predict oestrogen receptor (ER) status, a key clinical variable for breast cancer patients. We trained different architectures of our framework on 240 samples and benchmarked against graph learning via graph kernels. Propagation Attribute graph kernels achieved a class-balanced accuracy of 66.18% in the development set (N = 104) while GNNs achieved a class-balanced accuracy of 90.00% on the same set when using the best combination of graph convolution and pooling layers. We further validated this architecture in internal (N = 112) and external test sets from different institutions (N = 281 and N = 350), demonstrating the generalizability of the method. Our results suggest that MULTIPLAI captures important TME features with clinical importance. This is the first application of GNNs to this type of data and opens up new opportunities for predictive modelling of highly multiplexed images.


  • Graph neural networks
  • Highly multiplexed imaging
  • Imaging mass cytometry
  • Tumour microenvironment
  • Breast cancer

M. Crispin-Ortuzar and F. Markowetz—Shared senior authorship.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-87444-5_10
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-87444-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)
Fig. 1.


  1. Ali, H.R., et al.: Imaging mass cytometry and multiplatform genomics define the phenogenomic landscape of breast cancer. Nat. Cancer 1(2), 163–175 (2020).

  2. Borgwardt, K.M., Ghisu, M.E., Llinares-López, F., O’Bray, L., Rieck, B.: Graph kernels: state-of-the-art and future challenges. CoRR abs/2011.0 (2020).

  3. Bronstein, M.M., Bruna, J., Lecun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond Euclidean data, July 2017.

  4. Cao, W., Yan, Z., He, Z., He, Z.: A comprehensive survey on geometric deep learning. IEEE Access 8, 35929–35949 (2020).

    CrossRef  Google Scholar 

  5. Chang, Q., Ornatsky, O.I., Siddiqui, I., Loboda, A., Baranov, V.I., Hedley, D.W.: Imaging mass cytometry. Cytometry Part A 91(2), 160–169 (2017).

  6. Curtis, C., et al.: The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature 486(7403), 346–352 (2012).

  7. Giesen, C., et al.: Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat. Methods 11(4), 417–422 (2014).

  8. Hao, S.J., Wan, Y., Xia, Y.Q., Zou, X., Zheng, S.Y.: Size-based separation methods of circulating tumor cells, February 2018.

  9. Jackson, H.W., et al.: The single-cell pathology landscape of breast cancer. Nature 578(7796), 615–620 (2020).

  10. Jaume, G., et al.: Quantifying explainers of graph neural networks in computational pathology. Technical report (2021)

    Google Scholar 

  11. Jaume, G., et al.: Towards explainable graph representations in digital pathology. Technical report (2020)

    Google Scholar 

  12. Kamińska, K., et al.: The role of the cell-cell interactions in cancer progression. J. Cell. Mol. Medi. 19(2), 283–296 (2015).

  13. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, September 2016.

  14. Kong, K., et al.: FLAG: adversarial data augmentation for graph neural networks, October 2020.

  15. Kriege, N.M., Johansson, F.D., Morris, C.: A survey on graph kernels. Appl. Netw. Sci. 5(1), 6 (2020).

  16. Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. In: 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, November 2015.

  17. Neumann, M., Garnett, R., Bauckhage, C., Kersting, K.: Propagation kernels: efficient graph kernels from propagated information. Mach. Learn. 102(2), 209–245 (2016).

  18. Onitilo, A.A., Engel, J.M., Greenlee, R.T., Mukesh, B.N.: Breast cancer subtypes based on ER/PR and Her2 expression: comparison of clinicopathologic features and survival. Clin. Med. Res. 7(1–2), 4–13 (2009).

  19. Putti, T.C., et al.: Estrogen receptor-negative breast carcinomas: a review of morphology and immunophenotypical analysis. Mod. Pathol. 18(1), 26–35 (2005).

  20. Schapiro, D., et al.: histoCAT: analysis of cell phenotypes and interactions in multiplex image cytometry data. Nat. Methods 14(9), 873–876 (2017).

  21. Somarakis, A., Van Unen, V., Koning, F., Lelieveldt, B., Hollt, T.: ImaCytE: visual exploration of cellular micro-environments for imaging mass cytometry data. IEEE Trans. Vis. Comput. Graph. 27(1), 98–110 (2021).

  22. Turajlic, S., Swanton, C.: Implications of cancer evolution for drug development, July 2017.

  23. Wang, F., Tax, D.M.J.: Survey on the attention based RNN model and its applications in computer vision, January 2016.

  24. Whiteside, T.L.: The tumor microenvironment and its role in promoting tumor growth, October 2008. /pmc/articles/PMC3689267/

  25. Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv, October 2018.

  26. Zhao, T., Liu, Y., Neves, L., Woodford, O., Jiang, M., Shah, N.: Data augmentation for graph neural networks. arXiv, July 2020.

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Paula Martin-Gonzalez .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 5608 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Martin-Gonzalez, P., Crispin-Ortuzar, M., Markowetz, F. (2021). Predictive Modelling of Highly Multiplexed Tumour Tissue Images by Graph Neural Networks. In: , et al. Interpretability of Machine Intelligence in Medical Image Computing, and Topological Data Analysis and Its Applications for Medical Data. IMIMIC TDA4MedicalData 2021 2021. Lecture Notes in Computer Science(), vol 12929. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87443-8

  • Online ISBN: 978-3-030-87444-5

  • eBook Packages: Computer ScienceComputer Science (R0)