Abstract
Whole slide images (WSI) encompass a wealth of information about the tumor micro-environment, which holds prognostic value for patients’ survival. While significant progress has been made in predicting patients’ survival risks from WSI, existing studies often overlook the importance of incorporating multi-resolution and multi-scale histological image features, as well as their interactions, in the prediction process. This paper introduces the dual-stream context-aware (DSCA) model, which aims to enhance survival risk prediction by leveraging multi-resolution histological images and multi-scale feature maps, along with their contextual information. The DSCA model comprises three prediction branches: two ResNet50 branches that learn features from multi-resolution images, and one feature fusion branch that aggregates multi-scale features by exploring their interactions. The feature fusion branch of the DSCA model incorporates a mixed attention module, which performs adaptive spatial fusion to enhance the multi-scale feature maps. Subsequently, the self-attention mechanism is developed to learn contextual and interactive information from the enhanced feature maps. The ordinal Cox loss is employed to optimize the model for generating patch-level predictions. Patient-level predictions are obtained by mean-pooling patch-level results. Experimental results conducted on colorectal cancer cohorts demonstrate that the proposed DSCA model achieves significant improvements over state-of-the-art methods in survival prognosis.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cai, H., et al.: MIST: multiple instance learning network based on Swin transformer for whole slide image classification of colorectal adenomas. J. Pathol. 259(2), 125–135 (2023)
Chen, R.J., et al.: Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 339–349. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_33
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning, pp. 1597–1607. PMLR (2020)
Coudray, N., et al.: Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24(10), 1559–1567 (2018)
Dosovitskiy, A., et al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Geessink, O.G.F., et al.: Computer aided quantification of intratumoral stroma yields an independent prognosticator in rectal cancer. Cell. Oncol. 42(3), 331–341 (2019). https://doi.org/10.1007/s13402-019-00429-z
Huang, Z., Chai, H., Wang, R., Wang, H., Yang, Y., Wu, H.: Integration of patch features through self-supervised learning and transformer for survival analysis on whole slide images. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 561–570. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_54
Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International conference on machine learning, pp. 2127–2136. PMLR (2018)
Jiang, S., Suriawinata, A.A., Hassanpour, S.: MHAttnSurv: multi-head attention for survival prediction using whole-slide pathology images. Comput. Biol. Med. 158, 106883 (2023)
Laleh, N.G., Echle, A., Muti, H.S., Hewitt, K.J., Schulz, V., Kather, J.N.: Deep learning for interpretable end-to-end survival prediction in gastrointestinal cancer histopathology. In: COMPAY 2021: The Third MICCAI Workshop on Computational Pathology (2021)
Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318–14328 (2021)
Li, R., Yao, J., Zhu, X., Li, Y., Huang, J.: Graph CNN for survival analysis on whole slide pathological images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_20
Liu, S., Johns, E., Davison, A.J.: End-to-end multi-task learning with attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1871–1880 (2019)
Liu, S., Huang, D., Wang, Y.: Learning spatial fusion for single-shot object detection. arXiv preprint arXiv:1911.09516 (2019)
Lv, Z., Lin, Y., Yan, R., Wang, Y., Zhang, F.: TransSurv: transformer-based survival analysis model integrating histopathological images and genomic data for colorectal cancer. IEEE/ACM Transactions on Computational Biology and Bioinformatics (2022)
Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Nat. Acad. Sci. 115(13), E2970–E2979 (2018)
Roy, S., kumar Jain, A., Lal, S., Kini, J.: A study about color normalization methods for histopathology images. Micron 114, 42–61 (2018)
Schirris, Y., Gavves, E., Nederlof, I., Horlings, H.M., Teuwen, J.: DeepSMILE: contrastive self-supervised pre-training benefits MSI and HRD classification directly from H &E whole-slide images in colorectal and breast cancer. Med. Image Anal. 79, 102464 (2022)
Shao, W., Wang, T., Huang, Z., Han, Z., Zhang, J., Huang, K.: Weakly supervised deep ordinal cox model for survival prediction from whole-slide pathological images. IEEE Trans. Med. Imaging 40(12), 3739–3747 (2021)
Skrede, O.J., et al.: Deep learning for prediction of colorectal cancer outcome: a discovery and validation study. Lancet 395(10221), 350–360 (2020)
Srinidhi, C.L., Ciga, O., Martel, A.L.: Deep neural network models for computational histopathology: a survey. Med. Image Anal. 67, 101813 (2021)
Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P., Vaswani, A.: Bottleneck transformers for visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16519–16529 (2021)
Vaswani, A., et al.: Attention is all you need. Advances in neural information processing systems, vol. 30 (2017)
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: ECA-Net: efficient channel attention for deep convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11534–11542 (2020)
Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_1
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)
Zhao, K., et al.: Artificial intelligence quantified tumour-stroma ratio is an independent predictor for overall survival in resectable colorectal cancer. EBioMedicine 61, 103054 (2020)
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. 82102135), the Natural Science Foundation of Liaoning Province (Grant No. 2022-YGJC-36), and the Fundamental Research Funds for the Central Universities (Grant No. DUT22YG114, DUT21RC(3)038).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gao, J., Jin, S., Wang, R., Wang, M., Wang, T., Xu, H. (2024). Dual-Stream Context-Aware Neural Network for Survival Prediction from Whole Slide Images. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14434. Springer, Singapore. https://doi.org/10.1007/978-981-99-8549-4_1
Download citation
DOI: https://doi.org/10.1007/978-981-99-8549-4_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8548-7
Online ISBN: 978-981-99-8549-4
eBook Packages: Computer ScienceComputer Science (R0)