Abstract
Pathological image classification is a hot research topic in computer vision. Accurate classification of pathological images plays a vital role in disease diagnosis and prognosis prediction. However, the size of pathological images varies greatly, which brings great challenges to AI-assisted diagnosis. Convolutional Neural Network (CNN) has been widely used in pathological image classification tasks and achieved good results. However, it is difficult for CNN to handle images that are too large in size, it needs to compress excessively large pathological images, which will bring the loss of image information. To address these issues, we proposed a two-branch fusion model, named BiFusionNet, which combines CNN and Graph Neural Network (GNN). In the CNN branch, we employ the Densenet201 network to extract the feature of pathological images. For the GNN branch, we used PNA as the graph neural network, and the pathological images were processed into the graph data structure, and then input into the three-layer graph neural network layer to extract the graph feature. Finally, the convolutional feature and the graph feature were fused to realize a more accurate classification of pathological images. In order to make the two branches complement each other, Focal loss was added to the GNN branch based on Cross Entropy Loss, and the samples, which were difficult to classify, were set to higher weights. Our model achieved the supreme classification performance of 67.03% ± 2.04% on breast cancer dataset BRACS, and the best classification performance was 97.33% ± 1.25% in the CRA dataset of rectal cancer.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Saltz, J., Gupta, R., Hou, L.: Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23(1), 181–193 (2018)
Bejnordi, B.E., Veta, M., Van Diest, P.J.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318(22), 2199–2210 (2017)
Huang, G., Liu, Z., Van der Maaten, L.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Pati, P., Jaume, G., Fernandes, LA.: HACT-Net: a hierarchical cell-to-tissue graph neural network for histopathological image classification. arXiv arXiv:2007.00584 (2020)
Chen, R.J., et al.: Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 339–349. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_33
Simonyan K., Zisserman A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Tan M., Le Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Pati, P., Jaume, G., Foncubierta, A.: Hierarchical graph representations in digital pathology. arXiv arXiv:2102.11057 (2021)
Vaswani, A., Shazeer, N., Parmar, N.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
He, K., Chen, X., Xie, S.: Masked auto encoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009 (2022)
Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-ResNet and the impact of residual connections on learning. Proc. AAAI Conf. Artif. Intell. 31, 4278–7284 (2017)
Huang, G., Liu, S., Van der Maaten, L.: CondenseNet: an efficient DenseNet using learned group convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2752–2761 (2018)
He, K., Zhang, X., Ren, S.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Dosovitskiy, A., Beyer, L., Kolesnikov, A.: An image is worth 16 × 16 word: transformers for image recognition at scale. arXiv arXiv:2010.11929 (2021)
Wang, X., et al.: Transpath: Transformer-based self-supervised learning for histopathological image classification. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 186–195. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_18
Shaban, M., Awan, R., Fraz, M.M.: Context-aware convolutional neural network for grading of colorectal cancer histology images. arXiv arXiv:1907.09478 (2019)
Howard, A.G., Zhu, M., Chen, B.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Zhou, Y., Graham, S., Alemi Koohbanani, N.: CGC-Net: cell graph convolutional network for grading of colorectal cancer histology images. In: 2019 IEEE/ CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, South Korea, pp. 388–398. IEEE (2019)
Sudharshan, P.J., Petitjean, C., Spanhol, F.: Multiple instance learning for histopathological breast cancer image classification. Exp. Syst. Appl. 117, 103–111 (2019)
Zhao, Y., Yang, F, Fang, Y.: Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 4836–4845. IEEE (2020)
Shao, Z., Bian, H., Chen, Y.: TransMIL: transformer based correlated multiple instance learning for whole slide image classification. Adv. Neural. Inf. Process. Syst. 34, 2136–2147 (2021)
Vahadane, A., Peng, T., Sethi, A.: Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imaging 35(8), 1962–1971 (2016)
Lee, Y., Park, J.H., Oh, S.: Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nat. Biomed. Eng., 1–15. https://doi.org/10.1038/s41551-022-00923-0
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Song, S. et al. (2023). Dual Branch Fusion Network for Pathological Image Classification with Extreme Different Image Size. In: Lu, H., Blumenstein, M., Cho, SB., Liu, CL., Yagi, Y., Kamiya, T. (eds) Pattern Recognition. ACPR 2023. Lecture Notes in Computer Science, vol 14407. Springer, Cham. https://doi.org/10.1007/978-3-031-47637-2_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-47637-2_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47636-5
Online ISBN: 978-3-031-47637-2
eBook Packages: Computer ScienceComputer Science (R0)