Abstract
Supervised learning tasks such as cancer survival prediction from gigapixel whole slide images (WSIs) are a critical challenge in computational pathology that requires modeling complex features of the tumor microenvironment. These learning tasks are often solved with deep multi-instance learning (MIL) models that do not explicitly capture intratumoral heterogeneity. We develop a novel variance pooling architecture that enables a MIL model to incorporate intratumoral heterogeneity into its predictions. Two interpretability tools based on “representative patches” are illustrated to probe the biological signals captured by these models. An empirical study with 4,479 gigapixel WSIs from the Cancer Genome Atlas shows that adding variance pooling onto MIL frameworks improves survival prediction performance for five cancer types.
I. Carmichael and A.H. Song—These authors contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
When a patient has multiple WSIs, the patches are unioned across the WSIs.
- 2.
Without the attention weights this would be the quadratic form \(v_k^T \text {cov}(H)v_k\).
- 3.
Note the patient batch size is not equal to the number of summands in (5); there are between 0 and \({B \atopwithdelims ()2}\) summands depending on the number of comparable pairs.
References
Andor, N., et al.: Pan-cancer analysis of the extent and consequences of intratumor heterogeneity. Nat. Med. 22(1), 105–113 (2016)
Campanella, G., et al.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. (2019)
Carmichael, I., et al.: Joint and individual analysis of breast cancer histologic images and genomic covariates. Ann. Appl. Statist. 15(4), 1697–1722 (2021)
Chen, R.J., et al.: Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 339–349. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_33
Coudray, N., et al.: Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24(10), 1559–1567 (2018)
Courtiol, P., et al.: Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25(10), 1519–1525 (2019)
Couture, H.D., et al.: Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4(1), 1–8 (2018)
Faust, K., et al.: Unsupervised resolution of histomorphologic heterogeneity in renal cell carcinoma using a brain tumor-educated neural network. JCO Clin. Cancer Inf. 4, 811–821 (2020)
Guinney, J., et al.: The consensus molecular subtypes of colorectal cancer. Nat. Med. 21(11), 1350–1356 (2015)
Gurcan, M.N., Boucheron, L.E., Can, A., Madabhushi, A., Rajpoot, N.M., Yener, B.: Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2, 147–171 (2009)
Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: Proceedings of the 35th International Conference on Machine Learning. PMLR (2018)
Levine, D.A.: Integrated genomic characterization of endometrial carcinoma. Nature 497(7447), 67–73 (2013)
Liu, J., et al.: An integrated tcga pan-cancer clinical data resource to drive high-quality survival outcome analytics. Cell 173(2), 400–416 (2018)
Lu, C., et al.: Nuclear shape and orientation features from h &e images predict survival in early-stage estrogen receptor-positive breast cancers. Lab. Investig. 98(11), 1438–1448 (2018)
Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5(6), 555–570 (2021)
Luck, M., Sylvain, T., Cohen, J.P., Cardinal, H., Lodi, A., Bengio, Y.: Learning to rank for censored survival data. arXiv preprint arXiv:1806.01984 (2018)
Marusyk, A., Janiszewska, M., Polyak, K.: Intratumor heterogeneity: the Rosetta stone of therapy resistance. Cancer Cell 37(4), 471–484 (2020)
Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl. Acad. Sci. 115(13), E2970–E2979 (2018)
Raynaud, F., Mina, M., Tavernari, D., Ciriello, G.: Pan-cancer inference of intra-tumor heterogeneity reveals associations with different forms of genomic instability. PLoS Genet. 14(9), e1007669 (2018)
Steck, H., Krishnapuram, B., Dehing-Oberije, C., Lambin, P., Raykar, V.C.: On ranking in survival analysis: bounds on the concordance index. Adv. Neural Inf. Process. Syst. 20 (2007)
Wulczyn, E., et al.: Interpretable survival prediction for colorectal cancer using deep learning. NPJ Digit. Med. 4(1), 1–13 (2021)
Zadeh, S.G., Schmid, M.: Bias in cross-entropy-based training of deep survival networks. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 3126–3137 (2020)
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)
Zhao, Y., et al.: Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4837–4846 (2020)
Zhu, X., Yao, J., Zhu, F., Huang, J.: Wsisa: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7234–7242 (2017)
Acknowledgement
We thank Katherine Hoadley for helpful suggestions. This work was supported in part by internal funds from BWH Pathology, NIGMS R35GM138216 (F.M.), BWH President’s Fund, MGH Pathology, BWH Precision Medicine Program, Google Cloud Research Grant, Nvidia GPU Grant Program and funding from the Fredrick National Lab. R.C. was additionally supported by the NSF graduate research fellowship. T.Y.C. was additionally funded by the NIH National Cancer Institute (NCI) Ruth L. Kirschstein National Service Award, T32CA251062. The content is solely the responsibility of the authors and does not reflect the official views of the NIH, NIGMS, NCI, or NSF.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Carmichael, I., Song, A.H., Chen, R.J., Williamson, D.F.K., Chen, T.Y., Mahmood, F. (2022). Incorporating Intratumoral Heterogeneity into Weakly-Supervised Deep Learning Models via Variance Pooling. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13432. Springer, Cham. https://doi.org/10.1007/978-3-031-16434-7_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-16434-7_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16433-0
Online ISBN: 978-3-031-16434-7
eBook Packages: Computer ScienceComputer Science (R0)
