Abstract
Weakly supervised classification of whole slide images (WSIs) in digital pathology typically involves making slide-level predictions by aggregating predictions from embeddings extracted from multiple individual tiles. However, these embeddings can fail to capture valuable information contained within the individual cells in each tile. Here we describe an embedding extraction method that combines tile-level embeddings with a cell-level embedding summary. We validated the method using four hematoxylin and eosin stained WSI classification tasks: human epidermal growth factor receptor 2 status and estrogen receptor status in primary breast cancer, breast cancer metastasis in lymph node tissue, and cell of origin classification in diffuse large B-cell lymphoma. For all tasks, the new method outperformed embedding extraction methods that did not include cell-level representations. Using the publicly available HEROHE Challenge data set, the method achieved a state-of-the-art performance of 90% area under the receiver operating characteristic curve. Additionally, we present a novel model explainability method that could identify cells associated with different classification groups, thus providing supplementary validation of the classification model. This deep learning approach has the potential to provide morphological insights that may improve understanding of complex underlying tumor pathologies.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Neal, R.D., et al.: Is increased time to diagnosis and treatment in symptomatic cancer associated with poorer outcomes? Systematic review. Br. J. Cancer 112(Suppl 1), S92-107 (2015)
Henry, N.L., Hayes, D.F.: Cancer biomarkers. Mol. Oncol. 6(2), 140–146 (2012)
Park, J.E., Kim, H.S.: Radiomics as a quantitative imaging biomarker: practical considerations and the current standpoint in neuro-oncologic studies. Nucl. Med. Mol. Imaging 52(2), 99–108 (2018)
Lee, K., et al.: Deep learning of histopathology images at the single cell level. Front. Artif. Intell. 4, 754641 (2021)
Niazi, M.K.K., Parwani, A.V., Gurcan, M.N.: Digital pathology and artificial intelligence. Lancet Oncol. 20(5), e253–e261 (2019)
van der Laak, J., Litjens, G., Ciompi, F.: Deep learning in histopathology: the path to the clinic. Nat. Med. 27(5), 775–784 (2021)
Shao, Z., et al.: TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In: Advances in Neural Information Processing Systems, vol. 34 pp. 2136–2147 (2021)
Wang, Y., et al.: CWC-transformer: a visual transformer approach for compressed whole slide image classification. Neural Comput. Appl. (2023)
Lu, M.Y., Williamson, D.F.K., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5(6), 555–570 (2021)
Chen, R.J., et al.: Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16144–16155 (2022)
Kang, M., Song, H., Park, S., Yoo, D., Pereira, S.: Benchmarking self-supervised learning on diverse pathology datasets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3344–3354 (2023)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Grill, J.-B., et al.: Bootstrap your own latent - a new approach to self-supervised learning. Adv. Neural. Inf. Process. Syst. 33, 21271–21284 (2020)
Abbasi-Sureshjani, S., et al.: Molecular subtype prediction for breast cancer using H&E specialized backbone. In: MICCAI Workshop on Computational Pathology, pp. 1–9 (2021)
Tellez, D., et al.: Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Med. Image Anal. 58, 101544 (2019)
Litjens, G., et al.: H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience 7(6), giy065 (2018)
Collins, E., Achanta, R., Süsstrunk, S.: Deep feature factorization for concept discovery. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018, pp. 352–368. Springer International Publishing, Cham (2018)
Schmidt, U., Weigert, M., Broaddus, C., Myers, G.: Cell detection with star-convex polygons. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 265–273. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_30
Ilse, M., Tomczak, J.M., Welling, M.: Attention-based deep multiple instance learning. ArXiv abs/1802.04712 (2018)
Attention-based Deep Multiple Instance Learning. https://github.com/AMLab-Amsterdam/AttentionDeepMIL. Accessed 24 Feb 2023
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems (NeurIPS 2017), vol. 31, pp. 5998–6008 (2017)
Xiong, Y., et al.: Nyströmformer: a Nystöm-based algorithm for approximating self-attention. Proc. Conf. AAAI Artif. Intell. 35(16), 14138–14148 (2021)
Nyström Attention. https://github.com/lucidrains/nystrom-attention. Accessed 24 Feb 2023
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), abs/1412.6980 (2015)
Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2650–2658 (2015)
HEROHE ECDP2020. https://ecdp2020.grand-challenge.org/. Accessed 24 Feb 2023
Conde-Sousa, E., et al.: HEROHE challenge: predicting HER2 status in breast cancer from hematoxylin-eosin whole-slide imaging. J. Imaging 8(8) (2022)
National Cancer Institute GDC Data Portal. https://portal.gdc.cancer.gov/. Accessed 24 Feb 2023
CAMELYON17 Grand Challenge. https://camelyon17.grand-challenge.org/Data/. Accessed 24 Feb 2023
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017)
Acknowledgments
We thank the Roche Diagnostic Solutions and Genentech Research Pathology Core Laboratory staff for tissue procurement and immunohistochemistry verification. We thank the participants from the GOYA and CAVALLI trials. The results published here are in part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga. We thank Maris Skujevskis, Uwe Schalles and Darta Busa for their help in curating the datasets and the annotations and Amal Lahiani for sharing the tumor segmentation model used for generating the results on HEROHE. The study was funded by F. Hoffmann-La Roche AG, Basel, Switzerland and writing support was provided by Adam Errington PhD of PharmaGenesis Cardiff, Cardiff, UK and was funded by F. Hoffmann-La Roche AG.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gildenblat, J., Yüce, A., Abbasi-Sureshjani, S., Korski, K. (2023). Deep Cellular Embeddings: An Explainable Plug and Play Improvement for Feature Representation in Histopathology. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14225. Springer, Cham. https://doi.org/10.1007/978-3-031-43987-2_75
Download citation
DOI: https://doi.org/10.1007/978-3-031-43987-2_75
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43986-5
Online ISBN: 978-3-031-43987-2
eBook Packages: Computer ScienceComputer Science (R0)