Abstract
Breast cancer is the most prevalent cancer among women, and its diagnosis requires the accurate identification and classification of histological features for effective patient management. Artificial intelligence, particularly through deep learning, represents the next frontier in cancer diagnosis and management. Notably, the use of convolutional neural networks and emerging Vision Transformers (ViT) has been reported to automate pathologists’ tasks, including tumor detection and classification, in addition to improving the efficiency of pathology services. Deep learning applications have also been extended to the prediction of protein expression, molecular subtype, mutation status, therapeutic efficacy, and outcome prediction directly from hematoxylin and eosin-stained slides, bypassing the need for immunohistochemistry or genetic testing. This review explores the current status and prospects of deep learning in breast cancer diagnosis with a focus on whole-slide image analysis. Artificial intelligence applications are increasingly applied to many tasks in breast pathology ranging from disease diagnosis to outcome prediction, thus serving as valuable tools for assisting pathologists and supporting breast cancer management.
Similar content being viewed by others
References
Rakha EA, Toss M, Shiino S et al (2021) Current and future applications of artificial intelligence in pathology: a clinical perspective. J Clin Pathol 74:409–414
Niazi MKK, Parwani AV, Gurcan MN (2019) Digital pathology and artificial intelligence. Lancet Oncol 20:e253–e261
Springenberg M, Frommholz A, Wenzel M et al (2023) From modern CNNs to vision transformers: assessing the performance, robustness, and classification strategies of deep learning models in histopathology. Med Image Anal 87:102809
Ibrahim A, Gamble P, Jaroensri R et al (2020) Artificial intelligence in digital breast pathology: techniques and applications. Breast 49:267–273
Zhu J, Liu M, Li X (2022) Progress on deep learning in digital pathology of breast cancer: a narrative review. Gland Surg 11:751–766
WHO Classification of Tumors Editorial Board (2019) WHO Classification of Tumors. Breast tumors, 5th edn. World Health Organization, Geneva
Xue T, Chang H, Ren M et al (2023) Deep learning to automatically evaluate HER2 gene amplification status from fluorescence in situ hybridization images. Sci Rep 13:9746
Ghahremani P, Li Y, Kaufman A et al (2022) Deep learning-inferred multiplex ImmunoFluorescence for immunohistochemical image quantification. Nat Mach Intell 4:401–412
Gutman DA, Cobb J, Somanna D et al (2013) Cancer digital slide archive: an informatics resource to support integrated in silico analysis of TCGA pathology data. J Am Med Inform Assoc 20:1091–1098
Evans AJ, Bauer TW, Bui MM et al (2018) US food and drug administration approval of whole slide imaging for primary diagnosis: a key milestone is reached and new questions are raised. Arch Pathol Lab Med 142:1383–1387
Mori I (2022) Current status of whole slide image (WSI) standardization in Japan. Acta Histochem Cytochem 55:85–91
Howard FM, Dolezal J, Kochanny S et al (2021) The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat Commun 12:4423
Yao J, Zhu X, Jonnagaddala J et al (2020) Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med Image Anal 65:101789
Duggento A, Conti A, Mauriello A et al (2021) Deep computational pathology in breast cancer. Semin Cancer Biol 72:226–237
Lu MY, Chen TY, Williamson DFK et al (2021) AI-based pathology predicts origins for cancers of unknown primary. Nature 594:106–110
Deng S, Zhang X, Yan W et al (2020) Deep learning in digital pathology image analysis: a survey. Front Med 14:470–487
Campanella G, Hanna MG, Geneslaw L et al (2019) Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 25:1301–1309
Woo S, Park J, Lee J-Y et al (2018) CBAM: convolutional block attention module. arXiv:1807.06521v2
Dai Z, Liu H, Le QV et al (2021) CoAtNet: marrying convolution and attention for all data sizes. arXiv:2106.04803v2
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6
Szegedy C, Liu W, Jia Y et al (2014) Going deeper with convolutions. arXiv:1409.4842v1
He K, Zhang X, Ren S, et al (2015) Deep residual learning for image recognition. arXiv:1512.03385
Huang G, Liu Z, Maaten LD, et al (2016) Densely connected convolutional networks. arXiv:1608.06993
Girshick R, Donahue J, Darrell T, et al (2014) Rich feature hierarchies for accurate objection and semantic segmentation. arXiv:1311.2524v5
Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv:1505.04597v1
Chen J, Lu Y, Yu Q et al (2021) TransUNet: transformers make strong encoders for medical image segmentation. arXiv:2102.04306v1
Iqbal MS, Ahmad W, Alizadehsani R, et al (2022) Breast cancer dataset, classification and detection using deep learning. Healthcare (Basel) 10
Ghassemi M, Oakden-Rayner L, Beam AL (2021) The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health 3:e745–e750
Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929v2
Homeyer A, Geißler C, Schwen LO et al (2022) Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology. Mod Pathol 35:1759–1769
Qu L, Liu S, Liu X et al (2022) Towards label-efficient automatic diagnosis and analysis: a comprehensive survey of advanced deep learning-based weakly-supervised, semi-supervised and self-supervised techniques in histopathological image analysis. Phys Med Biol. https://doi.org/10.1088/1361-6560/ac910a
Wahab N, Miligy IM, Dodd K et al (2022) Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations. J Pathol Clin Res 8:116–128
Yousif M, van Diest PJ, Laurinavicius A et al (2022) Artificial intelligence applied to breast pathology. Virchows Arch 480:191–209
Gecer B, Aksoy S, Mercan E et al (2018) Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks. Pattern Recognit 84:345–356
Kanavati F, Tsuneki M (2021) Breast invasive ductal carcinoma classification on whole slide images with weakly-supervised and transfer learning. Cancers 13:5368
Mi W, Li J, Guo Y et al (2021) Deep learning-based multi-class classification of breast digital pathology images. Cancer Manag Res 13:4605–4617
Tripathi S, Singh SK, Lee HK (2021) An end-to-end breast tumour classification model using context-based patch modelling - A BiLSTM approach for image classification. Comput Med Imaging Graph 87:101838
Steiner DF, MacDonald R, Liu Y et al (2018) Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol 42:1636–1646
Liu Y, Kohlberger T, Norouzi M et al (2019) Artificial intelligence-based breast cancer nodal metastasis detection: insights into the black box for pathologists. Arch Pathol Lab Med 143:859–868
Kim YG, Song IH, Lee H et al (2020) Challenge for diagnostic assessment of deep learning algorithm for metastases classification in sentinel lymph nodes on frozen tissue section digital slides in women with breast cancer. Cancer Res Treat 52:1103–1111
Challa B, Tahir M, Hu Y et al (2023) Artificial intelligence-aided diagnosis of breast cancer lymph node metastasis on histologic slides in a digital workflow. Mod Pathol 36:100216
Balkenhol MCA, Tellez D, Vreuls W et al (2019) Deep learning assisted mitotic counting for breast cancer. Lab Invest 99:1596–1606
Pantanowitz L, Hartman D, Qi Y et al (2020) Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses. Diagn Pathol 15:80
Nateghi R, Danyali H, Helfroush MS (2021) A deep learning approach for mitosis detection: application in tumor proliferation prediction from whole slide images. Artif Intell Med 114:102048
Mantrala S, Ginter PS, Mitkari A et al (2022) Concordance in breast cancer grading by artificial intelligence on whole slide images compares with a multi-institutional cohort of breast pathologists. Arch Pathol Lab Med 146:1369–1377
Köteles MM, Vigdorovits A, Kumar D et al (2023) Comparative evaluation of breast ductal carcinoma grading: a deep-learning model and general pathologists’ assessment approach. Diagnostics 13:2326
Wetstein SC, Stathonikos N, Pluim JPW et al (2021) Deep learning-based grading of ductal carcinoma in situ in breast histopathology images. Lab Invest 101:525–533
Atallah NM, Wahab N, Toss MS et al (2023) Deciphering the morphology of tumor-stromal features in invasive breast cancer using artificial intelligence. Mod Pathol 36:100254
Yosofvand M, Khan SY, Dhakal R et al (2023) Automated detection and scoring of tumor-infiltrating lymphocytes in breast cancer histopathology slides. Cancers 15:3635
Makhlouf S, Wahab N, Toss M et al (2023) Evaluation of tumour infiltrating lymphocytes in luminal breast cancer using artificial intelligence. Br J Cancer 129:1747–1758
Shamai G, Binenbaum Y, Slossberg R et al (2019) Artificial intelligence algorithms to assess hormonal status from tissue microarrays in patients with breast cancer. JAMA Netw Open 2:e197700
Naik N, Madani A, Esteva A et al (2020) Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains. Nat Commun 11:5727
Khameneh FD, Razavi S, Kamasak M (2019) Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network. Comput Biol Med 110:164–174
Anand D, Kurian NC, Dhage S et al (2020) Deep learning to estimate human epidermal growth factor Receptor 2 status from hematoxylin and eosin-stained breast tissue images. J Pathol Inform 11:19
Rawat RR, Ortega I, Roy P et al (2020) Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images. Sci Rep 10:7275
Shamai G, Livne A, Polónia A et al (2022) Deep learning-based image analysis predicts PD-L1 status from H&E-stained histology images in breast cancer. Nat Commun 13:6753
Couture HD, Williams LA, Geradts J et al (2018) Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4:30
Kather JN, Heij LR, Grabsch HI et al (2020) Pan-cancer image-based detection of clinically actionable genetic alterations. Nat Cancer 1:789–799
Jaber MI, Song B, Taylor C et al (2020) A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival. Breast Cancer Res 22:12
Liu H, Xu WD, Shang ZH et al (2022) Breast cancer molecular subtype prediction on pathological images with discriminative patch selection and multi-instance learning. Front Oncol 12:858453
Mondol RK, Millar EKA, Graham PH et al (2023) hist2RNA: an efficient deep learning architecture to predict gene expression from breast cancer histopathology images. Cancers 15:2569
Qu H, Zhou M, Yan Z et al (2021) Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning. NPJ Precis Oncol 5:87
Wang X, Zou C, Zhang Y et al (2021) Prediction of BRCA gene mutation in breast cancer based on deep learning and histopathology images. Front Genet 12:661109
Morel LO, Derangère V, Arnould L et al (2023) Preliminary evaluation of deep learning for first-line diagnostic prediction of tumor mutational status. Sci Rep 13:6927
Cheerla A, Gevaert O (2019) Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35:i446–i454
Bychkov D, Joensuu H, Nordling S et al (2022) Outcome and biomarker supervised deep learning for survival prediction in two multicenter breast cancer series. J Pathol Inform 13:9
Wang Y, Acs B, Robertson S et al (2022) Improved breast cancer histological grading using deep learning. Ann Oncol 33:89–98
Wetstein SC, de Jong VMT, Stathonikos N et al (2022) Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images. Sci Rep 12:15102
Jaroensri R, Wulczyn E, Hegde N et al (2022) Deep learning models for histologic grading of breast cancer and association with disease prognosis. NPJ Breast Cancer 8:113
Wahab N, Toss M, Miligy IM et al (2023) AI-enabled routine H&E image based prognostic marker for early-stage luminal breast cancer. NPJ Precis Oncol 7:122
Su Z, Niazi MKK, Tavolara TE et al (2023) BCR-Net: a deep learning framework to predict breast cancer recurrence from histopathology images. PLoS One 18:e0283562
Fan J, Zhang L, Lv T et al (2023) MEAI: an artificial intelligence platform for predicting distant and lymph node metastases directly from primary breast cancer. J Cancer Res Clin Oncol 149:9229–9241
Bychkov D, Linder N, Tiulpin A et al (2021) Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy. Sci Rep 11:4037
Li F, Yang Y, Wei Y et al (2021) Deep learning-based predictive biomarker of pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer. J Transl Med 19:348
Li B, Li F, Liu Z et al (2022) Deep learning with biopsy whole slide images for pretreatment prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer: a multicenter study. Breast 66:183–190
Huang Z, Shao W, Han Z et al (2023) Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images. NPJ Precis Oncol 7:14
Hida AI, Omanovic D, Pedersen L et al (2020) Automated assessment of Ki-67 in breast cancer: the utility of digital image analysis using virtual triple staining and whole slide imaging. Histopathology 77:471–480
Shafi S, Kellough DA, Lujan G et al (2022) Integrating and validating automated digital imaging analysis of estrogen receptor immunohistochemistry in a fully digital workflow for clinical use. J Pathol Inform 13:100122
Aung TN, Acs B, Warrell J et al (2021) A new tool for technical standardization of the Ki67 immunohistochemical assay. Mod Pathol 34:1261–1270
Sandbank J, Bataillon G, Nudelman A et al (2022) Validation and real-world clinical application of an artificial intelligence algorithm for breast cancer detection in biopsies. NPJ Breast Cancer 8:129
Saednia K, Tran WT, Sadeghi-Naini A (2023) A hierarchical self-attention-guided deep learning framework to predict breast cancer response to chemotherapy using pre-treatment tumor biopsies. Med Phys 50:7852–7864
Wang Z, Gao Q, Yi X et al (2023) Surformer: an interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images. Comput Methods Programs Biomed 241:107733
Tuli S, Dasgupta I, Grant E, et al (2021) Are convolutional neural networks or transformers more like human vision? arXiv:2105.07197v2
Acknowledgements
We thank the Kurozumi Medical Foundation for supporting AK. This study was also supported by Gunma University through a Priority Support program Grant.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Katayama, A., Aoki, Y., Watanabe, Y. et al. Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers. Int J Clin Oncol (2024). https://doi.org/10.1007/s10147-024-02513-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10147-024-02513-3