Skip to main content
SpringerLink
Log in

Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers

  • Invited Review Article
  • Published:
International Journal of Clinical Oncology Aims and scope Submit manuscript

Abstract

Breast cancer is the most prevalent cancer among women, and its diagnosis requires the accurate identification and classification of histological features for effective patient management. Artificial intelligence, particularly through deep learning, represents the next frontier in cancer diagnosis and management. Notably, the use of convolutional neural networks and emerging Vision Transformers (ViT) has been reported to automate pathologists’ tasks, including tumor detection and classification, in addition to improving the efficiency of pathology services. Deep learning applications have also been extended to the prediction of protein expression, molecular subtype, mutation status, therapeutic efficacy, and outcome prediction directly from hematoxylin and eosin-stained slides, bypassing the need for immunohistochemistry or genetic testing. This review explores the current status and prospects of deep learning in breast cancer diagnosis with a focus on whole-slide image analysis. Artificial intelligence applications are increasingly applied to many tasks in breast pathology ranging from disease diagnosis to outcome prediction, thus serving as valuable tools for assisting pathologists and supporting breast cancer management.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Rakha EA, Toss M, Shiino S et al (2021) Current and future applications of artificial intelligence in pathology: a clinical perspective. J Clin Pathol 74:409–414

    Article  CAS  PubMed  Google Scholar 

  2. Niazi MKK, Parwani AV, Gurcan MN (2019) Digital pathology and artificial intelligence. Lancet Oncol 20:e253–e261

    Article  PubMed  PubMed Central  Google Scholar 

  3. Springenberg M, Frommholz A, Wenzel M et al (2023) From modern CNNs to vision transformers: assessing the performance, robustness, and classification strategies of deep learning models in histopathology. Med Image Anal 87:102809

    Article  PubMed  Google Scholar 

  4. Ibrahim A, Gamble P, Jaroensri R et al (2020) Artificial intelligence in digital breast pathology: techniques and applications. Breast 49:267–273

    Article  PubMed  Google Scholar 

  5. Zhu J, Liu M, Li X (2022) Progress on deep learning in digital pathology of breast cancer: a narrative review. Gland Surg 11:751–766

    Article  PubMed  PubMed Central  Google Scholar 

  6. WHO Classification of Tumors Editorial Board (2019) WHO Classification of Tumors. Breast tumors, 5th edn. World Health Organization, Geneva

    Google Scholar 

  7. Xue T, Chang H, Ren M et al (2023) Deep learning to automatically evaluate HER2 gene amplification status from fluorescence in situ hybridization images. Sci Rep 13:9746

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Ghahremani P, Li Y, Kaufman A et al (2022) Deep learning-inferred multiplex ImmunoFluorescence for immunohistochemical image quantification. Nat Mach Intell 4:401–412

    Article  PubMed  PubMed Central  Google Scholar 

  9. Gutman DA, Cobb J, Somanna D et al (2013) Cancer digital slide archive: an informatics resource to support integrated in silico analysis of TCGA pathology data. J Am Med Inform Assoc 20:1091–1098

    Article  PubMed  PubMed Central  Google Scholar 

  10. Evans AJ, Bauer TW, Bui MM et al (2018) US food and drug administration approval of whole slide imaging for primary diagnosis: a key milestone is reached and new questions are raised. Arch Pathol Lab Med 142:1383–1387

    Article  PubMed  Google Scholar 

  11. Mori I (2022) Current status of whole slide image (WSI) standardization in Japan. Acta Histochem Cytochem 55:85–91

    Article  PubMed  PubMed Central  Google Scholar 

  12. Howard FM, Dolezal J, Kochanny S et al (2021) The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat Commun 12:4423

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Yao J, Zhu X, Jonnagaddala J et al (2020) Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med Image Anal 65:101789

    Article  PubMed  Google Scholar 

  14. Duggento A, Conti A, Mauriello A et al (2021) Deep computational pathology in breast cancer. Semin Cancer Biol 72:226–237

    Article  PubMed  Google Scholar 

  15. Lu MY, Chen TY, Williamson DFK et al (2021) AI-based pathology predicts origins for cancers of unknown primary. Nature 594:106–110

    Article  CAS  PubMed  Google Scholar 

  16. Deng S, Zhang X, Yan W et al (2020) Deep learning in digital pathology image analysis: a survey. Front Med 14:470–487

    Article  PubMed  Google Scholar 

  17. Campanella G, Hanna MG, Geneslaw L et al (2019) Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 25:1301–1309

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Woo S, Park J, Lee J-Y et al (2018) CBAM: convolutional block attention module. arXiv:1807.06521v2

  19. Dai Z, Liu H, Le QV et al (2021) CoAtNet: marrying convolution and attention for all data sizes. arXiv:2106.04803v2

  20. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6

  21. Szegedy C, Liu W, Jia Y et al (2014) Going deeper with convolutions. arXiv:1409.4842v1

  22. He K, Zhang X, Ren S, et al (2015) Deep residual learning for image recognition. arXiv:1512.03385

  23. Huang G, Liu Z, Maaten LD, et al (2016) Densely connected convolutional networks. arXiv:1608.06993

  24. Girshick R, Donahue J, Darrell T, et al (2014) Rich feature hierarchies for accurate objection and semantic segmentation. arXiv:1311.2524v5

  25. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv:1505.04597v1

  26. Chen J, Lu Y, Yu Q et al (2021) TransUNet: transformers make strong encoders for medical image segmentation. arXiv:2102.04306v1

  27. Iqbal MS, Ahmad W, Alizadehsani R, et al (2022) Breast cancer dataset, classification and detection using deep learning. Healthcare (Basel) 10

  28. Ghassemi M, Oakden-Rayner L, Beam AL (2021) The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health 3:e745–e750

    Article  CAS  PubMed  Google Scholar 

  29. Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929v2

  30. Homeyer A, Geißler C, Schwen LO et al (2022) Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology. Mod Pathol 35:1759–1769

    Article  PubMed  PubMed Central  Google Scholar 

  31. Qu L, Liu S, Liu X et al (2022) Towards label-efficient automatic diagnosis and analysis: a comprehensive survey of advanced deep learning-based weakly-supervised, semi-supervised and self-supervised techniques in histopathological image analysis. Phys Med Biol. https://doi.org/10.1088/1361-6560/ac910a

    Article  PubMed  Google Scholar 

  32. Wahab N, Miligy IM, Dodd K et al (2022) Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations. J Pathol Clin Res 8:116–128

    Article  PubMed  PubMed Central  Google Scholar 

  33. Yousif M, van Diest PJ, Laurinavicius A et al (2022) Artificial intelligence applied to breast pathology. Virchows Arch 480:191–209

    Article  PubMed  Google Scholar 

  34. Gecer B, Aksoy S, Mercan E et al (2018) Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks. Pattern Recognit 84:345–356

    Article  PubMed  PubMed Central  Google Scholar 

  35. Kanavati F, Tsuneki M (2021) Breast invasive ductal carcinoma classification on whole slide images with weakly-supervised and transfer learning. Cancers 13:5368

    Article  PubMed  PubMed Central  Google Scholar 

  36. Mi W, Li J, Guo Y et al (2021) Deep learning-based multi-class classification of breast digital pathology images. Cancer Manag Res 13:4605–4617

    Article  PubMed  PubMed Central  Google Scholar 

  37. Tripathi S, Singh SK, Lee HK (2021) An end-to-end breast tumour classification model using context-based patch modelling - A BiLSTM approach for image classification. Comput Med Imaging Graph 87:101838

    Article  PubMed  Google Scholar 

  38. Steiner DF, MacDonald R, Liu Y et al (2018) Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol 42:1636–1646

    Article  PubMed  PubMed Central  Google Scholar 

  39. Liu Y, Kohlberger T, Norouzi M et al (2019) Artificial intelligence-based breast cancer nodal metastasis detection: insights into the black box for pathologists. Arch Pathol Lab Med 143:859–868

    Article  CAS  PubMed  Google Scholar 

  40. Kim YG, Song IH, Lee H et al (2020) Challenge for diagnostic assessment of deep learning algorithm for metastases classification in sentinel lymph nodes on frozen tissue section digital slides in women with breast cancer. Cancer Res Treat 52:1103–1111

    PubMed  PubMed Central  Google Scholar 

  41. Challa B, Tahir M, Hu Y et al (2023) Artificial intelligence-aided diagnosis of breast cancer lymph node metastasis on histologic slides in a digital workflow. Mod Pathol 36:100216

    Article  PubMed  Google Scholar 

  42. Balkenhol MCA, Tellez D, Vreuls W et al (2019) Deep learning assisted mitotic counting for breast cancer. Lab Invest 99:1596–1606

    Article  PubMed  Google Scholar 

  43. Pantanowitz L, Hartman D, Qi Y et al (2020) Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses. Diagn Pathol 15:80

    Article  PubMed  PubMed Central  Google Scholar 

  44. Nateghi R, Danyali H, Helfroush MS (2021) A deep learning approach for mitosis detection: application in tumor proliferation prediction from whole slide images. Artif Intell Med 114:102048

    Article  PubMed  Google Scholar 

  45. Mantrala S, Ginter PS, Mitkari A et al (2022) Concordance in breast cancer grading by artificial intelligence on whole slide images compares with a multi-institutional cohort of breast pathologists. Arch Pathol Lab Med 146:1369–1377

    Article  CAS  PubMed  Google Scholar 

  46. Köteles MM, Vigdorovits A, Kumar D et al (2023) Comparative evaluation of breast ductal carcinoma grading: a deep-learning model and general pathologists’ assessment approach. Diagnostics 13:2326

    Article  PubMed  PubMed Central  Google Scholar 

  47. Wetstein SC, Stathonikos N, Pluim JPW et al (2021) Deep learning-based grading of ductal carcinoma in situ in breast histopathology images. Lab Invest 101:525–533

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Atallah NM, Wahab N, Toss MS et al (2023) Deciphering the morphology of tumor-stromal features in invasive breast cancer using artificial intelligence. Mod Pathol 36:100254

    Article  PubMed  Google Scholar 

  49. Yosofvand M, Khan SY, Dhakal R et al (2023) Automated detection and scoring of tumor-infiltrating lymphocytes in breast cancer histopathology slides. Cancers 15:3635

    Article  PubMed  PubMed Central  Google Scholar 

  50. Makhlouf S, Wahab N, Toss M et al (2023) Evaluation of tumour infiltrating lymphocytes in luminal breast cancer using artificial intelligence. Br J Cancer 129:1747–1758

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Shamai G, Binenbaum Y, Slossberg R et al (2019) Artificial intelligence algorithms to assess hormonal status from tissue microarrays in patients with breast cancer. JAMA Netw Open 2:e197700

    Article  PubMed  PubMed Central  Google Scholar 

  52. Naik N, Madani A, Esteva A et al (2020) Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains. Nat Commun 11:5727

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Khameneh FD, Razavi S, Kamasak M (2019) Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network. Comput Biol Med 110:164–174

    Article  CAS  PubMed  Google Scholar 

  54. Anand D, Kurian NC, Dhage S et al (2020) Deep learning to estimate human epidermal growth factor Receptor 2 status from hematoxylin and eosin-stained breast tissue images. J Pathol Inform 11:19

    Article  PubMed  PubMed Central  Google Scholar 

  55. Rawat RR, Ortega I, Roy P et al (2020) Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images. Sci Rep 10:7275

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  56. Shamai G, Livne A, Polónia A et al (2022) Deep learning-based image analysis predicts PD-L1 status from H&E-stained histology images in breast cancer. Nat Commun 13:6753

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  57. Couture HD, Williams LA, Geradts J et al (2018) Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4:30

    Article  PubMed  PubMed Central  Google Scholar 

  58. Kather JN, Heij LR, Grabsch HI et al (2020) Pan-cancer image-based detection of clinically actionable genetic alterations. Nat Cancer 1:789–799

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Jaber MI, Song B, Taylor C et al (2020) A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival. Breast Cancer Res 22:12

    Article  PubMed  PubMed Central  Google Scholar 

  60. Liu H, Xu WD, Shang ZH et al (2022) Breast cancer molecular subtype prediction on pathological images with discriminative patch selection and multi-instance learning. Front Oncol 12:858453

    Article  PubMed  PubMed Central  Google Scholar 

  61. Mondol RK, Millar EKA, Graham PH et al (2023) hist2RNA: an efficient deep learning architecture to predict gene expression from breast cancer histopathology images. Cancers 15:2569

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  62. Qu H, Zhou M, Yan Z et al (2021) Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning. NPJ Precis Oncol 5:87

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Wang X, Zou C, Zhang Y et al (2021) Prediction of BRCA gene mutation in breast cancer based on deep learning and histopathology images. Front Genet 12:661109

    Article  PubMed  PubMed Central  Google Scholar 

  64. Morel LO, Derangère V, Arnould L et al (2023) Preliminary evaluation of deep learning for first-line diagnostic prediction of tumor mutational status. Sci Rep 13:6927

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Cheerla A, Gevaert O (2019) Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35:i446–i454

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Bychkov D, Joensuu H, Nordling S et al (2022) Outcome and biomarker supervised deep learning for survival prediction in two multicenter breast cancer series. J Pathol Inform 13:9

    Article  PubMed  Google Scholar 

  67. Wang Y, Acs B, Robertson S et al (2022) Improved breast cancer histological grading using deep learning. Ann Oncol 33:89–98

    Article  CAS  PubMed  Google Scholar 

  68. Wetstein SC, de Jong VMT, Stathonikos N et al (2022) Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images. Sci Rep 12:15102

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  69. Jaroensri R, Wulczyn E, Hegde N et al (2022) Deep learning models for histologic grading of breast cancer and association with disease prognosis. NPJ Breast Cancer 8:113

    Article  PubMed  PubMed Central  Google Scholar 

  70. Wahab N, Toss M, Miligy IM et al (2023) AI-enabled routine H&E image based prognostic marker for early-stage luminal breast cancer. NPJ Precis Oncol 7:122

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  71. Su Z, Niazi MKK, Tavolara TE et al (2023) BCR-Net: a deep learning framework to predict breast cancer recurrence from histopathology images. PLoS One 18:e0283562

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  72. Fan J, Zhang L, Lv T et al (2023) MEAI: an artificial intelligence platform for predicting distant and lymph node metastases directly from primary breast cancer. J Cancer Res Clin Oncol 149:9229–9241

    Article  PubMed  Google Scholar 

  73. Bychkov D, Linder N, Tiulpin A et al (2021) Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy. Sci Rep 11:4037

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  74. Li F, Yang Y, Wei Y et al (2021) Deep learning-based predictive biomarker of pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer. J Transl Med 19:348

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Li B, Li F, Liu Z et al (2022) Deep learning with biopsy whole slide images for pretreatment prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer: a multicenter study. Breast 66:183–190

    Article  PubMed  PubMed Central  Google Scholar 

  76. Huang Z, Shao W, Han Z et al (2023) Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images. NPJ Precis Oncol 7:14

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  77. Hida AI, Omanovic D, Pedersen L et al (2020) Automated assessment of Ki-67 in breast cancer: the utility of digital image analysis using virtual triple staining and whole slide imaging. Histopathology 77:471–480

    Article  PubMed  Google Scholar 

  78. Shafi S, Kellough DA, Lujan G et al (2022) Integrating and validating automated digital imaging analysis of estrogen receptor immunohistochemistry in a fully digital workflow for clinical use. J Pathol Inform 13:100122

    Article  PubMed  PubMed Central  Google Scholar 

  79. Aung TN, Acs B, Warrell J et al (2021) A new tool for technical standardization of the Ki67 immunohistochemical assay. Mod Pathol 34:1261–1270

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Sandbank J, Bataillon G, Nudelman A et al (2022) Validation and real-world clinical application of an artificial intelligence algorithm for breast cancer detection in biopsies. NPJ Breast Cancer 8:129

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  81. Saednia K, Tran WT, Sadeghi-Naini A (2023) A hierarchical self-attention-guided deep learning framework to predict breast cancer response to chemotherapy using pre-treatment tumor biopsies. Med Phys 50:7852–7864

    Article  CAS  PubMed  Google Scholar 

  82. Wang Z, Gao Q, Yi X et al (2023) Surformer: an interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images. Comput Methods Programs Biomed 241:107733

    Article  PubMed  Google Scholar 

  83. Tuli S, Dasgupta I, Grant E, et al (2021) Are convolutional neural networks or transformers more like human vision? arXiv:2105.07197v2

Download references

Acknowledgements

We thank the Kurozumi Medical Foundation for supporting AK. This study was also supported by Gunma University through a Priority Support program Grant.

Author information

Authors and Affiliations

  1. Diagnostic Pathology, Gunma University Graduate School of Medicine, 3-39-22 Showamachi, Maebashi, Gunma, 371-8511, Japan

    Ayaka Katayama & Tetsunari Oyama

  2. Center for Mathematics and Data Science, Gunma University, Maebashi, Japan

    Yuki Aoki

  3. Clinical Training Center, Gunma University Hospital, Maebashi, Japan

    Yukako Watanabe

  4. Department of Breast Surgery, International University of Health and Welfare, Narita, Japan

    Jun Horiguchi

  5. Department of Histopathology School of Medicine, University of Nottingham, University Park, Nottingham, UK

    Emad A. Rakha

  6. Department of Pathology, Hamad Medical Corporation, Doha, Qatar

    Emad A. Rakha

Authors
  1. Ayaka Katayama
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuki Aoki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yukako Watanabe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Horiguchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emad A. Rakha
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tetsunari Oyama
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayaka Katayama.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Katayama, A., Aoki, Y., Watanabe, Y. et al. Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers. Int J Clin Oncol (2024). https://doi.org/10.1007/s10147-024-02513-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10147-024-02513-3

Keywords

Search

Navigation