Skip to main content
SpringerLink
Log in

Deep learning in digital pathology image analysis: a survey

  • Review
  • Published:
Frontiers of Medicine Aims and scope Submit manuscript

Abstract

Deep learning (DL) has achieved state-of-the-art performance in many digital pathology analysis tasks. Traditional methods usually require hand-crafted domain-specific features, and DL methods can learn representations without manually designed features. In terms of feature extraction, DL approaches are less labor intensive compared with conventional machine learning methods. In this paper, we comprehensively summarize recent DL-based image analysis studies in histopathology, including different tasks (e.g., classification, semantic segmentation, detection, and instance segmentation) and various applications (e.g., stain normalization, cell/gland/region structure analysis). DL methods can provide consistent and accurate outcomes. DL is a promising tool to assist pathologists in clinical diagnosis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cardesa A, Zidar N, Alos L, Nadal A, Gale N, Klöppel G. The Kaiser’s cancer revisited: was Virchow totally wrong? Virchows Arch 2011; 458(6): 649–657

    PubMed  Google Scholar 

  2. Gurcan MN, Boucheron LE, Can A, Madabhushi A, Rajpoot NM, Yener B. Histopathological image analysis: a review. IEEE Rev Biomed Eng 2009; 2: 147–171

    PubMed  PubMed Central  Google Scholar 

  3. Andrion A, Magnani C, Betta PG, Donna A, Mollo F, Scelsi M, Bernardi P, Botta M, Terracini B. Malignant mesothelioma of the pleura: interobserver variability. J Clin Pathol 1995; 48(9): 856–860

    CAS  PubMed  PubMed Central  Google Scholar 

  4. Wolberg WH, Street WN, Heisey DM, Mangasarian OL. Computer-derived nuclear features distinguish malignant from benign breast cytology. Hum Pathol 1995; 26(7): 792–796

    CAS  PubMed  Google Scholar 

  5. Thiran JP, Macq B. Morphological feature extraction for the classification of digital images of cancerous tissues. IEEE Trans Biomed Eng 1996; 43(10): 1011–1020

    CAS  PubMed  Google Scholar 

  6. Choi HK, Jarkrans T, Bengtsson E, Vasko J, Wester K, Malmström PU, Busch C. Image analysis based grading of bladder carcinoma. Comparison of object, texture and graph based methods and their reproducibility. Anal Cell Pathol 1997; 15(1): 1–18

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Hamilton PW, Bartels PH, Thompson D, Anderson NH, Montironi R, Sloan JM. Automated location of dysplastic fields in colorectal histology using image texture analysis. J Pathol 1997; 182(1): 68–75

    CAS  PubMed  Google Scholar 

  8. Esgiar AN, Naguib RN, Sharif BS, Bennett MK, Murray A. Fractal analysis in the detection of colonic cancer images. IEEE Trans Inf Technol Biomed 2002; 6(1): 54–58

    PubMed  Google Scholar 

  9. Spyridonos P, Ravazoula P, Cavouras D, Berberidis K, Nikiforidis G. Computer-based grading of haematoxylin-eosin stained tissue sections of urinary bladder carcinomas. Med Inform Internet Med 2001; 26(3): 179–190

    CAS  PubMed  Google Scholar 

  10. Wiltgen M, Gerger A, Smolle J. Tissue counter analysis of benign common nevi and malignant melanoma. Int J Med Inform 2003; 69 (1): 17–28

    CAS  PubMed  Google Scholar 

  11. Nielsen B, Albregtsen F, Danielsen HE. The use of fractal features from the periphery of cell nuclei as a classification tool. Anal Cell Pathol 1999; 19(1): 21–37

    CAS  PubMed  PubMed Central  Google Scholar 

  12. Esgiar AN, Naguib RN, Sharif BS, Bennett MK, Murray A. Microscopic image analysis for quantitative measurement and feature identification of normal and cancerous colonic mucosa. IEEE Trans Inf Technol Biomed 1998; 2(3): 197–203

    CAS  PubMed  Google Scholar 

  13. Weyn B, van de Wouwer G, Kumar-Singh S, van Daele A, Scheunders P, van Marck E, Jacob W. Computer-assisted differential diagnosis of malignant mesothelioma based on syntactic structure analysis. Cytometry 1999; 35(1): 23–29

    CAS  PubMed  Google Scholar 

  14. Keenan SJ, Diamond J, McCluggage WG, Bharucha H, Thompson D, Bartels PH, Hamilton PW. An automated machine vision system for the histological grading of cervical intraepithelial neoplasia (CIN). J Pathol 2000; 192(3): 351–362

    CAS  PubMed  Google Scholar 

  15. Demir C, Gultekin SH, Yener B. Learning the topological properties of brain tumors. IEEE/ACM Trans Comput Biol Bioinformatics 2005; 2(3): 262–270

    Google Scholar 

  16. Gunduz-Demir C. Mathematical modeling of the malignancy of cancer using graph evolution. Math Biosci 2007; 209(2): 514–527

    PubMed  Google Scholar 

  17. Weinstein RS, Graham AR, Richter LC, Barker GP, Krupinski EA, Lopez AM, Erps KA, Bhattacharyya AK, Yagi Y, Gilbertson JR. Overview of telepathology, virtual microscopy, and whole slide imaging: prospects for the future. Hum Pathol 2009; 40(8): 1057–1069

    PubMed  Google Scholar 

  18. Krupinski EA, Bhattacharyya AK, Weinstein RS. Telepathology and Digital Pathology Research. Springer, Cham. 2016: 41–54

    Google Scholar 

  19. Farahani N, Parwani AV, Pantanowitz L. Whole slide imaging in pathology: advantages, limitations, and emerging perspectives. Pathol Lab Med Int 2015; 7: 23–33

    Google Scholar 

  20. Ying X, Monticello TM. Modern imaging technologies in toxicologic pathology: an overview. Toxicol Pathol 2006; 34(7): 815–826

    PubMed  Google Scholar 

  21. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521 (7553): 436–444

    CAS  PubMed  Google Scholar 

  22. LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD. Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems. 1990: 396–404

  23. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998; 86(11): 2278–2324

    Google Scholar 

  24. Wang P, Ge R, Xiao X, Cai Y, Wang G, Zhou F. Rectified-linear-unit-based deep learning for biomedical multi-label data. Inter-discip Sci 2017; 9(3): 419–422

    Google Scholar 

  25. Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med Image Anal 2016; 33: 170–175

    PubMed  PubMed Central  Google Scholar 

  26. Epstein JI. An update of the Gleason grading system. J Urol 2010; 183(2): 433–440

    PubMed  Google Scholar 

  27. Frierson HF Jr, Wolber RA, Berean KW, Franquemont DW, Gaffey MJ, Boyd JC, Wilbur DC. Interobserver reproducibility of the Nottingham modification of the Bloom and Richardson histologic grading scheme for infiltrating ductal carcinoma. Am J Clin Pathol 1995; 103(2): 195–198

    PubMed  Google Scholar 

  28. Khan AM, Rajpoot N, Treanor D, Magee D. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans Biomed Eng 2014; 61(6): 1729–1738

    PubMed  Google Scholar 

  29. Vahadane A, Peng T, Sethi A, Albarqouni S, Wang L, Baust M, Steiger K, Schlitter AM, Esposito I, Navab N. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans Med Imaging 2016; 35(8): 1962–1971

    PubMed  Google Scholar 

  30. Janowczyk A, Basavanhally A, Madabhushi A. Stain normalization using Sparse AutoEncoders (StaNoSA): application to digital pathology. Comput Med Imaging Graph 2017; 57: 50–61

    PubMed  Google Scholar 

  31. Bentaieb A, Hamarneh G. Adversarial stain transfer for histopathology image analysis. IEEE Trans Med Imaging 2018; 37(3): 792–802

    PubMed  Google Scholar 

  32. Roy S, Kumar Jain A, Lal S, Kini J. A study about color normalization methods for histopathology images. Micron 2018; 114: 42–61

    PubMed  Google Scholar 

  33. Elston CW, Ellis IO. Author Commentary: “Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up. C. W. Elston & I. O. Ellis. Histopathology 1991; 19; 403–410.” Histopathology 2002; 41(3A): 151

    CAS  PubMed  Google Scholar 

  34. Chow KH, Factor RE, Ullman KS. The nuclear envelope environment and its cancer connections. Nat Rev Cancer 2012; 12(3): 196–209

    CAS  PubMed  PubMed Central  Google Scholar 

  35. Dey P. Cancer nucleus: morphology and beyond. Diagn Cytopathol 2010; 38(5): 382–390

    PubMed  Google Scholar 

  36. Chen H, Wang X, Heng PA. Automated mitosis detection with deep regression networks. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE. 2016: 1204–1207

  37. Cireşan D, Giusti A, Gambardella LM, Schmidhuber J. Deep neural networks segment neuronal membranes in electron microscopy images. Adv Neural Inf Process Syst 2012: 2843–2851

  38. Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2015: 234–241

    Google Scholar 

  39. Zhou Y, Chang H, Barner KE, Parvin B. Nuclei segmentation via sparsity constrained convolutional regression. Proc IEEE Int Symp Biomed Imaging 2015; 2015: 1284–1287

    PubMed  PubMed Central  Google Scholar 

  40. Song Y, Zhang L, Chen S, Ni D, Lei B, Wang T. Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning. IEEE Trans Biomed Eng 2015; 62(10): 2421–2433

    PubMed  Google Scholar 

  41. Xing F, Xie Y, Yang L. An automatic learning-based framework for robust nucleus segmentation. IEEE Trans Med Imaging 2016; 35(2): 550–566

    PubMed  Google Scholar 

  42. Pan X, Li L, Yang H, Liu Z, Yang J, Zhao L, Fan Y. Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing 2016; 229: S0925231216313765

    Google Scholar 

  43. Zhang L, Sonka M, Lu L, Summers RM, Yao J. Combining fully convolutional networks and graph-based approach for automated segmentation of cervical cell nuclei. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). IEEE. 2017: 406–409

  44. Alom MZ, Yakopcic C, Taha TM, Asari VK. Nuclei segmentation with recurrent residual convolutional neural networks based U-Net (R2U-Net). NAECON 2018-IEEE National Aerospace and Electronics Conference. IEEE. 2018: 228–233

  45. Xie Y, Xing F, Kong X, Su H, Yang L. Beyond classification: structured regression for robust cell detection using convolutional neural network. Med Image Comput Comput Assist Interv 2015; 9351: 358–365

    PubMed  PubMed Central  Google Scholar 

  46. Sirinukunwattana K, Ahmed Raza SE, Tsang YW, Snead DRJ, Cree IA, Rajpoot NM. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 2016; 35(5): 1196–1206

    PubMed  Google Scholar 

  47. Xu J, Xiang L, Liu Q, Gilmore H, Wu J, Tang J, Madabhushi A. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans Med Imaging 2016; 35 (1): 119–130

    PubMed  Google Scholar 

  48. Xie Y, Xing F, Shi X, Kong X, Su H, Yang L. Efficient and robust cell detection: a structured regression approach. Med Image Anal 2018; 44: 245–254

    PubMed  Google Scholar 

  49. Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J. Mitosis detection in breast cancer histology images with deep neural networks. Med Image Comput Comput Assist Interv 2013; 16(Pt 2): 411–418

    PubMed  Google Scholar 

  50. Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, Tomaszewski J, Gonzalez F, Madabhushi A. Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J Med Imaging (Bellingham) 2014; 1(3): 034003

    Google Scholar 

  51. Chen H, Dou Q, Wang X, Qin J, Heng PA. Mitosis detection in breast cancer histology images via deep cascaded networks. Thirtieth AAAI Conference on Artificial Intelligence. 2016

  52. Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S, Navab N. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans Med Imaging 2016; 35(5): 1313–1321

    PubMed  Google Scholar 

  53. Li C, Wang X, Liu W, Latecki LJ. DeepMitosis: mitosis detection via deep detection, verification and segmentation networks. Med Image Anal 2018; 45: 121–133

    PubMed  Google Scholar 

  54. Ma M, Shi Y, Li W, Gao Y, Xu J. A novel two-stage deep method for mitosis detection in breast cancer histology images. 2018 24th International Conference on Pattern Recognition (ICPR). IEEE. 2018: 3892–3897

  55. Li C, Wang X, Liu W, Latecki LJ, Wang B, Huang J. Weakly supervised mitosis detection in breast histopathology images using concentric loss. Med Image Anal 2019; 53: 165–178

    PubMed  Google Scholar 

  56. Yang L, Zhang Y, Guldner IH, Zhang S, Chen DZ. 3D segmentation of glial cells using fully convolutional networks and k-terminal cut. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 658–666

    Google Scholar 

  57. Akram SU, Kannala J, Eklund L, Heikkilä J. Cell proposal network for microscopy image analysis. 2016 IEEE International Conference on Image Processing (ICIP). IEEE. 2016: 3199–3203

  58. Akram SU, Kannala J, Eklund L, Heikkilä J. Cell segmentation proposal network for microscopy image analysis. Deep Learning and Data Labeling for Medical Applications. Springer, Cham. 2016: 21–29

    Google Scholar 

  59. Chen H, Qi X, Yu L, Dou Q, Qin J, Heng PA. DCAN: deep contour-aware networks for object instance segmentation from histology images. Med Image Anal 2017; 36: 135–146

    PubMed  Google Scholar 

  60. Song Y, Tan EL, Jiang X, Cheng JZ, Ni D, Chen S, Lei B, Wang T. Accurate cervical cell segmentation from overlapping clumps in pap smear images. IEEE Trans Med Imaging 2017; 36(1): 288–300

    PubMed  Google Scholar 

  61. Kumar N, Verma R, Sharma S, Bhargava S, Vahadane A, Sethi A. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans Med Imaging 2017; 36(7): 1550–1560

    PubMed  Google Scholar 

  62. Ho DJ, Fu C, Salama P, Dunn KW, Delp EJ. Nuclei detection and segmentation of fluorescence microscopy images using three dimensional convolutional neural networks. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE. 2018: 418–422

  63. Naylor P, Laé M, Reyal F, Walter T. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Trans Med Imaging 2019; 38(2): 448–459

    PubMed  Google Scholar 

  64. Zhou Y, Onder OF, Dou Q, Tsougenis E, Chen H, Heng PA. Cianet: Robust nuclei instance segmentation with contour-aware information aggregation. International Conference on Information Processing in Medical Imaging. Springer, Cham. 2019: 682–693

    Google Scholar 

  65. Arganda-Carreras I, Seung HS, Cardona A, Schindelin J. Segmentation of neuronal structures in EM stacks challenge-ISBI 2012. 2012

  66. Oren A, Fernandes J. The Bethesda system for the reporting of cervical/vaginal cytology. J Am Osteopath Assoc 1991; 91(5): 476–479

    CAS  PubMed  Google Scholar 

  67. Naik S, Doyle S, Feldman M, Tomaszewski J, Madabhushi A. Gland segmentation and computerized gleason grading of prostate histology by integrating low-, high-level and domain specific information. MIAAB workshop. 2007: 1–8

  68. Karvelis PS, Fotiadis DI, Georgiou I, Syrrou M. A watershed based segmentation method for multispectral chromosome images classification. Conf Proc IEEE Eng Med Biol Soc 2006; 2006: 3009–3012

    PubMed  Google Scholar 

  69. Petushi S, Garcia FU, Haber MM, Katsinis C, Tozeren A. Large-scale computations on histology images reveal grade-differentiating parameters for breast cancer. BMC Med Imaging 2006; 6(1): 14

    PubMed  PubMed Central  Google Scholar 

  70. Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 2017; 39(4): 640–651

    PubMed  Google Scholar 

  71. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B, Rueckert D. Attention U-net: learning where to look for the pancreas. arXiv 2018; 1804.03999

  72. Zeng Z, Xie W, Zhang Y, Lu Y. RIC-Unet: an improved neural network based on Unet for nuclei segmentation in histology images. IEEE Access 2019; 7: 21420–21428

    Google Scholar 

  73. Chen JM, Li Y, Xu J, Gong L, Wang LW, Liu WL, Liu J. Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: a review. Tumour Biol 2017; 39(3): 1010428317694550

    Google Scholar 

  74. Veta M, Pluim JP, van Diest PJ, Viergever MA. Breast cancer histopathology image analysis: a review. IEEE Trans Biomed Eng 2014; 61(5): 1400–1411

    PubMed  Google Scholar 

  75. Sommer C, Fiaschi L, Hamprecht FA, Gerlich DW. Learning-based mitotic cell detection in histopathological images. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE. 2012: 2306–2309

  76. Veta M, van Diest PJ, Pluim JPW. Detecting mitotic figures in breast cancer histopathology images. Medical Imaging 2013: Digital Pathology. International Society for Optics and Photonics. 2013; 8676: 867607

    Google Scholar 

  77. Khan AM, Eldaly H, Rajpoot NM. A gamma-Gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. J Pathol Inform 2013; 4(4): 149–152

    Google Scholar 

  78. Paul A, Dey A, Mukherjee DP, Sivaswamy J, Tourani V. Regenerative random forest with automatic feature selection to detect mitosis in histopathological breast cancer images. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2015: 94–102

    Google Scholar 

  79. Cruz-Roa AA, Arevalo Ovalle JE, Madabhushi A, González Osorio FA. A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. Med Image Comput Comput Assist Interv 2013; 16(Pt 2): 403–410

    PubMed  Google Scholar 

  80. Veta M, van Diest PJ, Willems SM, Wang H, Madabhushi A, Cruz-Roa A, Gonzalez F, Larsen AB, Vestergaard JS, Dahl AB, Cireşan DC, Schmidhuber J, Giusti A, Gambardella LM, Tek FB, Walter T, Wang CW, Kondo S, Matuszewski BJ, Precioso F, Snell V, Kittler J, de Campos TE, Khan AM, Rajpoot NM, Arkoumani E, Lacle MM, Viergever MA, Pluim JP. Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 2015; 20(1): 237–248

    PubMed  Google Scholar 

  81. Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Naour GL, Gurcan MN. Mitosis detection in breast cancer histological images. An ICPR 2012 contest. J Pathol Inform 2013; 4: 8

    PubMed  PubMed Central  Google Scholar 

  82. Yang F, Mackey MA, Ianzini F, Gallardo G, Sonka M. Cell segmentation, tracking, and mitosis detection using temporal context. Med Image Comput Comput Assist Interv 2005; 8(Pt 1): 302–309

    PubMed  Google Scholar 

  83. Payer C, Štern D, Feiner M, Bischof H, Urschler M. Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks. Med Image Anal 2019; 57: 106–119

    PubMed  Google Scholar 

  84. Hariharan B, Arbeláez P, Girshick R, Malik J. Simultaneous detection and segmentation. European Conference on Computer Vision. Springer, Cham. 2014: 297–312

    Google Scholar 

  85. Veta M, Van Diest PJ, Pluim JPW. Cutting out the middleman: measuring nuclear area in histopathology slides without segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 632–639

    Google Scholar 

  86. Kronqvist P, Kuopio T, Collan Y. Morphometric grading of invasive ductal breast cancer. I. Thresholds for nuclear grade. Br J Cancer 1998; 78(6): 800–805

    CAS  PubMed  PubMed Central  Google Scholar 

  87. Mommers EC, Page DL, Dupont WD, Schuyler P, Leonhart AM, Baak JP, Meijer CJ, van Diest PJ. Prognostic value of morphometry in patients with normal breast tissue or usual ductal hyperplasia of the breast. Int J Cancer 2001; 95(5): 282–285

    CAS  PubMed  Google Scholar 

  88. Veta M, Kornegoor R, Huisman A, Verschuur-Maes AH, Viergever MA, Pluim JP, van Diest PJ. Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer. Mod Pathol 2012; 25(12): 1559–1565

    PubMed  Google Scholar 

  89. Maška M, Ulman V, Svoboda D, Matula P, Matula P, Ederra C, Urbiola A, España T, Venkatesan S, Balak DMW, Karas P, Bolcková T, Štreitová M, Carthel C, Coraluppi S, Harder N, Rohr K, Magnusson KEG, Jaldén J, Blau HM, Dzyubachyk O, Křížek P, Hagen GM, Escuredo DP, Carretero DJ, Carbayo MJL, Barrutia AM, Meijering E, Kozubek M, Solorzano CO. A benchmark for comparison of cell tracking algorithms. Bioinformatics 2014; 30 (11): 1609–1617

    PubMed  PubMed Central  Google Scholar 

  90. Vu QD, Graham S, To NN. Minh, Muhammad S, Talha Q, Navid AK, Ali KS, Tahsin K, Keyvan F, Tianhao Z, Rajarsi G, Tae KJ, Nasir R, Joel S. Methods for segmentation and classification of digital microscopy tissue images. Front Bioeng Biotechnol 2019; 7: 53

    PubMed  PubMed Central  Google Scholar 

  91. Naylor P, Laé M, Reyal F, et al. Nuclei segmentation in histopathology images using deep neural networks[C]//2017 IEEE 14th International Symposium On Biomedical Imaging (ISBI 2017). IEEE. 2017: 933–936

  92. Kumar N, Verma R, Anand D, Zhou Y, Onder OF, Tsougenis E, Chen H, Heng P, Li J, Hu Z, Wang Y, Koohbanani NA, Jahanifar M, Tajeddin NZ, Gooya A, Rajpoot N, Ren X, Zhou S, Wang Q, Shen D, Yang C, Weng C, Yu W, Yeh C, Yang S, Xu S, Yeung PH, Sun P, Mahbod A, Schaefer G, Ellinger I, Ecker R, Smedby O, Wang C, Chidester B, Ton T, Tran M, Ma J, Do MN, Graham S, Vu QD, Kwak JT, Gunda A, Chunduri R, Hu C, Zhou X, Lotfi D, Safdari R, Kascenas A, O’Neil A, Eschweiler D, Stegmaier J, Cui Y, Yin B, Chen K, Tian X, Gruening P, Barth E, Arbel E, Remer L, Ben-Dor A, Sirazitdinova E, Kohl M, Braunewell S, Li Y, Xie X, Shen L, Ma J, Baksi KD, Khan MA, Choo J, Colomer A, Naranjo V, Pei L, Iftekharuddin KM, Roy K, Bhattacharjee D, Pedraza A, Bueno MG, Devanathan S, Radhakrishnan S, Koduganty P, Wu Z, Cai G, Liu X, Wang Y, Sethi A. A multi-organ nucleus segmentation challenge. IEEE Trans Med Imaging 2020; 39(5): 1380–1391

    PubMed  Google Scholar 

  93. Neubeck A, Van Gool L. Efficient non-maximum suppression. 18th International Conference on Pattern Recognition (ICPR’06). IEEE. 2006, 3: 850–855

  94. Maurer CR, Qi R, Raghavan V. A linear time algorithm for computing exact euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans Pattern Anal Mach Intell 2003; 25(2): 265–270

    Google Scholar 

  95. Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 2017; 39(6): 1137–1149

    PubMed  Google Scholar 

  96. Zhang R, Cheng C, Zhao X, Li X. Multiscale mask R-CNN-based lung tumor detection using PET imaging. Mol Imaging 2019; 18: 1536012119863531

    PubMed  PubMed Central  Google Scholar 

  97. Wang S, Rong R, Yang DM, Fujimoto J, Yan S, Cai L, Yang L, Luo D, Behrens C, Parra ER, Yao B, Xu L, Wang T, Zhan X, Wistuba II, Minna J, Xie Y, Xiao G. Computational staining of pathology images to study the tumor microenvironment in lung cancer. Cancer Res 2020; 80(10): 2056–2066

    PubMed  Google Scholar 

  98. Johnson JW. Adapting mask-RCNN for automatic nucleus segmentation. arXiv 2018; 1805.00500

  99. Hoda SA, Hoda RS. Rubin’s pathology: clinicopathologic foundations of medicine. JAMA 2007; 298(17): 2070–2075

    Google Scholar 

  100. Sirinukunwattana K, Pluim JPW, Chen H, Qi X, Heng PA, Guo YB, Wang LY, Matuszewski BJ, Bruni E, Sanchez U, Böhm A, Ronneberger O, Cheikh BB, Racoceanu D, Kainz P, Pfeiffer M, Urschler M, Snead DRJ, Rajpoot NM. Gland segmentation in colon histology images: the GlaS challenge contest. Med Image Anal 2017; 35: 489–502

    PubMed  Google Scholar 

  101. Gunduz-Demir C, Kandemir M, Tosun AB, Sokmensuer C. Automatic segmentation of colon glands using object-graphs. Med Image Anal 2010; 14(1): 1–12

    PubMed  Google Scholar 

  102. Hess KR, Varadhachary GR, Taylor SH, Wei W, Raber MN, Lenzi R, Abbruzzese JL. Metastatic patterns in adenocarcinoma. Cancer 2006; 106(7): 1624–1633

    PubMed  Google Scholar 

  103. Ryan DP, Hong TS, Bardeesy N. Pancreatic adenocarcinoma. N Engl J Med 2014; 371(11): 1039–1049

    CAS  PubMed  Google Scholar 

  104. Fleming M, Ravula S, Tatishchev SF, Wang HL. Colorectal carcinoma: pathologic aspects. J Gastrointest Oncol 2012; 3(3): 153–173

    PubMed  PubMed Central  Google Scholar 

  105. Travis WD, Brambilla E, Geisinger KR. Histological grading in lung cancer: one system for all or separate systems for each histological type? Eur Respir J 2016; 47(3): 720–723

    PubMed  Google Scholar 

  106. Xu Y, Li Y, Liu M, Wang Y, Lai M, Eric C. Gland instance segmentation by deep multichannel side supervision. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 496–504

    Google Scholar 

  107. Kainz P, Pfeiffer M, Urschler M. Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization. PeerJ 2017; 5: e3874

    PubMed  PubMed Central  Google Scholar 

  108. Li W, Manivannan S, Akbar S, Zhang J, Trucco E, McKenna SJ. Gland segmentation in colon histology images using hand-crafted features and convolutional neural networks. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE. 2016: 1405–1408

  109. Xu Y, Li Y, Wang Y, Liu M, Fan Y, Lai M, Chang EIC. Gland instance segmentation using deep multichannel neural networks. IEEE Trans Biomed Eng 2017; 64(12): 2901–2912

    PubMed  Google Scholar 

  110. BenTaieb A, Hamarneh G. Topology aware fully convolutional networks for histology gland segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 460–468

    Google Scholar 

  111. BenTaieb A, Kawahara J, Hamarneh G. Multi-loss convolutional networks for gland analysis in microscopy. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE. 2016: 642–645

  112. Wu HS, Xu R, Harpaz N, Burstein D, Gil J. Segmentation of microscopic images of small intestinal glands with directional 2-D filters. Anal Quant Cytol Histol 2005; 27(5): 291–300

    CAS  PubMed  Google Scholar 

  113. Farjam R, Soltanian-Zadeh H, Jafari-Khouzani K, Zoroofi RA. An image analysis approach for automatic malignancy determination of prostate pathological images. Cytometry B Clin Cytom 2007; 72 (4): 227–240

    PubMed  Google Scholar 

  114. Wu HS, Xu R, Harpaz N, Burstein D, Gil J. Segmentation of intestinal gland images with iterative region growing. J Microsc 2005; 220(Pt 3): 190–204

    PubMed  Google Scholar 

  115. Fu H, Qiu G, Shu J, Ilyas M. A novel polar space random field model for the detection of glandular structures. IEEE Trans Med Imaging 2014; 33(3): 764–776

    PubMed  Google Scholar 

  116. Sirinukunwattana K, Snead DR, Rajpoot NM. A stochastic polygons model for glandular structures in colon histology images. IEEE Trans Med Imaging 2015; 34(11): 2366–2378

    PubMed  Google Scholar 

  117. Monaco JP, Tomaszewski JE, Feldman MD, Hagemann I, Moradi M, Mousavi P, Boag A, Davidson C, Abolmaesumi P, Madabhushi A. High-throughput detection of prostate cancer in histological sections using probabilistic pairwise Markov models. Med Image Anal 2010; 14(4): 617–629

    PubMed  PubMed Central  Google Scholar 

  118. Diamond J, Anderson NH, Bartels PH, Montironi R, Hamilton PW. The use of morphological characteristics and texture analysis in the identification of tissue composition in prostatic neoplasia. Hum Pathol 2004; 35(9): 1121–1131

    PubMed  Google Scholar 

  119. Doyle S, Madabhushi A, Feldman M, Tomaszeweski J. A boosting cascade for automated detection of prostate cancer from digitized histology. Med Image Comput Comput Assist Interv 2006; 9(Pt 2): 504–511

    PubMed  Google Scholar 

  120. Tabesh A, Teverovskiy M, Pang HY, Kumar VP, Verbel D, Kotsianti A, Saidi O. Multifeature prostate cancer diagnosis and Gleason grading of histological images. IEEE Trans Med Imaging 2007; 26(10): 1366–1378

    PubMed  Google Scholar 

  121. Nguyen K, Sarkar A, Jain AK. Structure and context in prostatic gland segmentation and classification. Med Image Comput Comput Assist Interv 2012; 15(Pt 1): 115–123

    PubMed  Google Scholar 

  122. Jacobs JG, Panagiotaki E, Alexander DC. Gleason grading of prostate tumours with max-margin conditional random fields. International Workshop on Machine Learning in Medical Imaging. Springer, Cham. 2014: 85–92

  123. Sabata B, Babenko B, Monroe R, Srinivas C. Automated analysis of pin-4 stained prostate needle biopsies. International Workshop on Prostate Cancer Imaging. Springer, Berlin, Heidelberg. 2010: 89–100

    Google Scholar 

  124. Altunbay D, Cigir C, Sokmensuer C, Gunduz-Demir C. Color graphs for automated cancer diagnosis and grading. IEEE Trans Biomed Eng 2010; 57(3): 665–674

    PubMed  Google Scholar 

  125. Fakhrzadeh A, Sporndly-Nees E, Holm L, Hendriks CLL. Analyzing tubular tissue in histopathological thin sections. 2012 International Conference on Digital Image Computing Techniques and Applications (DICTA). IEEE. 2012: 1–6

  126. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 2012: 1097–1105

  127. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. 2015: 1–9

  128. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248–255

  129. Manivannan S, Li W, Akbar S, Wang R, Zhang J, McKenna SJ. An automated pattern recognition system for classifying indirect immunofluorescence images of hep-2 cells and specimens. Pattern Recognit 2016; 51: 12–26

    Google Scholar 

  130. Xu Y, Jia Z, Wang LB, Ai Y, Zhang F, Lai M, Chang EIC. Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinformatics 2017; 18(1): 281

    PubMed  PubMed Central  Google Scholar 

  131. Xu Y, Mo T, Feng Q, Zhong P, Lai M, Chang EI. Deep learning of feature representation with multiple instance learning for medical image analysis. IEEE International Conference on Acoustics, Speech and Signal Processing 2014; 1626–1630

  132. Xu Y, Jia Z, Ai Y, Zhang F, Lai M, Change EI. Deep convolutional activation features for large scale brain tumor histopathology image classification and segmentation. IEEE International Conference on Acoustics, Speech and Signal Processing. 2015: 947–951

  133. Hou L, Samaras D, Kurc TM, Gao Y, Davis JE, Saltz JH. Patch-based convolutional neural network for whole slide tissue image classification. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2016; 2016: 2424–2433

    PubMed  PubMed Central  Google Scholar 

  134. Källén H, Molin J, Heyden A, Lundstrom C, Astrom K. Towards grading gleason score using generically trained deep convolutional neural networks. IEEE International Symposium on Biomedical Imaging. 2016: 1163–1167

  135. Xu J, Luo X, Wang G, Gilmore H, Madabhushi A. A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 2016; 191: 214–223

    PubMed  PubMed Central  Google Scholar 

  136. Qaiser T, Tsang YW, Epstein D, Rajpoot N. Tumor segmentation in whole slide images using persistent homology and deep convolutional features. Annual Conference on Medical Image Understanding and Analysis. 2017; 723: 320–329

    Google Scholar 

  137. Jia Z, Huang X, Chang EI, Xu Y. Constrained deep weak supervision for histopathology image segmentation. IEEE Trans Med Imaging 2017; 36(11): 2376–2388

    PubMed  Google Scholar 

  138. Courtiol P, Tramel EW, Sanselme M, Wainrib G. Classification and disease localization in histopathology using only global labels: a weakly-supervised approach. 2018

  139. Wang X, Chen H, Gan C, Lin H, Dou Q, Tsougenis E, Huang Q, Cai M, Heng PA. Weakly supervised deep learning for whole slide lung cancer image analysis. IEEE Trans Cybern 2019; [Epub ahead of print] doi: https://doi.org/10.1109/TCYB.2019.2935141

  140. Mercan C, Aksoy S, Mercan E, Shapiro LG, Weaver DL, Elmore JG. From patch-level to roi-level deep feature representations for breast histopathology classification. Medical Imaging 2019: Digital Pathology. 2019. 109560H

  141. Xu Y, Jiao L, Wang S, Wei J, Fan Y, Lai M, Chang EI. Multi-label classification for colon cancer using histopathological images. Microsc Res Tech 2013; 76(12): 1266–1277

    PubMed  Google Scholar 

  142. Jiao L, Qi C, Li S, Yan X. Colon cancer detection using whole slide histopathological images. IFMBE Proc 2013; 39: 1283–1286

    Google Scholar 

  143. Adankon MM, Cheriet M. Support vector machine. Comput Sci 2002; 1(4): 1–28

    Google Scholar 

  144. Golland P, Hata N, Barillot C, Hornegger J, Howe R. Preface. Medical image computing and computer-assisted intervention—MICCAI 2014. Med Image Comput Comput Assist Interv 2014; 17(Pt 1): V–VI

    PubMed  Google Scholar 

  145. Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, Lecun Y. Overfeat: integrated recognition, localization and detection using convolutional networks. International Conference on Learning Representations. 2014

  146. Liaw A, Wiener M. Classification and regression by random forest. R News 2007; 2: 18–22

    Google Scholar 

  147. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv 2014; 1409.1556v6

  148. Yan X, Zhu JY, Chang EI, Tu Z. Multiple clustered instance learning for histopathology cancer image classification, segmentation and clustering. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2012; 964–971

  149. Xu Y, Zhang J, Chang EI, Lai M, Tu Z. Context-constrained multiple instance learning for histopathology image segmentation. Med Image Comput Comput Assist Interv 2012; 15(Pt 3): 623–630

    PubMed  Google Scholar 

  150. Xu Y, Zhu JY, Chang EI, Lai M, Tu Z. Weakly supervised histopathology cancer image segmentation and classification. Med Image Anal 2014; 18(3): 591–604

    PubMed  Google Scholar 

  151. Xu Y, Li Y, Shen Z, Wu Z, Gao T, Fan Y, Lai M, Chang EI. Parallel multiple instance learning for extremely large histopathology image analysis. BMC Bioinformatics 2017; 18(1): 360

    CAS  PubMed  PubMed Central  Google Scholar 

  152. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2016: 770–778

  153. Hagerty RG, Butow PN, Ellis PM, Dimitry S, Tattersall MH. Communicating prognosis in cancer care: a systematic review of the literature. Ann Oncol 2005; 16(7): 1005–1053

    CAS  PubMed  Google Scholar 

  154. Norgeot B, Glicksberg BS, Butte AJ. A call for deep-learning healthcare. Nat Med 2019; 25(1): 14–15

    CAS  PubMed  Google Scholar 

  155. Uramoto H, Tanaka F. Recurrence after surgery in patients with NSCLC. Transl Lung Cancer Res 2014; 3(4): 242–249

    PubMed  PubMed Central  Google Scholar 

  156. Wang X, Janowczyk A, Zhou Y, Thawani R, Fu P, Schalper K, Velcheti V, Madabhushi A. Prediction of recurrence in early stage non-small cell lung cancer using computer extracted nuclear features from digital H&E images. Sci Rep 2017; 7(1): 13543

    PubMed  PubMed Central  Google Scholar 

  157. Vaidya P, Wang X, Bera K, Khunger A, Choi H, Patil P, Velcheti V, Madabhushi A. Raptomics: integrating radiomic and pathomic features for predicting recurrence in early stage lung cancer. Medical Imaging 2018: Digital Pathology International Society for Optics and Photonics. 2018: 105810M

  158. Sanchez-Cespedes M, Parrella P, Esteller M, Nomoto S, Trink B, Engles JM, Westra WH, Herman JG, Sidransky D. Inactivation of LKB1/STK11 is a common event in adenocarcinomas of the lung. Cancer Res 2002; 62(13): 3659–3662

    CAS  PubMed  Google Scholar 

  159. Shackelford DB, Abt E, Gerken L, Vasquez DS, Seki A, Leblanc M, Wei L, Fishbein MC, Czernin J, Mischel PS, Shaw RJ. LKB1 inactivation dictates therapeutic response of non-small cell lung cancer to the metabolism drug phenformin. Cancer Cell 2013; 23 (2): 143–158

    CAS  PubMed  PubMed Central  Google Scholar 

  160. Parums DV. Current status of targeted therapy in non-small cell lung cancer. Drugs Today (Barc) 2014; 50(7): 503–525

    CAS  Google Scholar 

  161. Wang S, Yang DM, Rong R, Zhan X, Fujimoto J, Liu H, Minna J, Wistuba II, Xie Y, Xiao G. Artificial intelligence in lung cancer pathology image analysis. Cancers (Basel) 2019; 11(11): 1673

    Google Scholar 

  162. Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, Moreira AL, Razavian N, Tsirigos A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med 2018; 24(10): 1559–1567

    CAS  PubMed  Google Scholar 

  163. Kather JN, Pearson AT, Halama N, Jäger D, Krause J, Loosen SH, Marx A, Boor P, Tacke F, Neumann UP, Grabsch HI, Yoshikawa T, Brenner H, Chang-Claude J, Hoffmeister M, Trautwein C, Luedde T. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 2019; 25(7): 1054–1056

    CAS  PubMed  PubMed Central  Google Scholar 

  164. Le DT, Uram JN, Wang H, Bartlett BR, Kemberling H, Eyring AD, Skora AD, Luber BS, Azad NS, Laheru D, Biedrzycki B, Donehower RC, Zaheer A, Fisher GA, Crocenzi TS, Lee JJ, Duffy SM, Goldberg RM, de la Chapelle A, Koshiji M, Bhaijee F, Huebner T, Hruban RH, Wood LD, Cuka N, Pardoll DM, Papadopoulos N, Kinzler KW, Zhou S, Cornish TC, Taube JM, Anders RA, Eshleman JR, Vogelstein B, Diaz LA Jr. Pd-1 blockade in tumors with mismatch-repair deficiency. N Engl J Med 2015; 372(26): 2509–2520

    CAS  PubMed  PubMed Central  Google Scholar 

  165. Nagpal K, Foote D, Liu Y, Chen PHC, Wulczyn E, Tan F, Olson N, Smith JL, Mohtashamian A, Wren JH, Corrado GS, MacDonald R, Peng LH, Amin MB, Evans AJ, Sangoi AR, Mermel CH, Hipp JD, Stumpe MC. Development and validation of a deep learning algorithm for improving gleason scoring of prostate cancer. NPJ Digit Med 2019; 2(1): 1–10

    Google Scholar 

  166. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2016: 2818–2826

  167. Ing N, Ma Z, Li J, Salemi H, Arnold C, Knudsen BS, Gertych A. Semantic segmentation for prostate cancer grading by convolutional neural networks. Medical Imaging 2018: Digital Pathology International Society for Optics and Photonics. 2018: 105811B

  168. Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 2017; 39(12): 2481–2495

    PubMed  Google Scholar 

  169. Li W, Li J, Sarma KV, Ho KC, Shen S, Knudsen BS, Gertych A, Arnold CW. Path R-CNN for prostate cancer diagnosis and gleason grading of histological images. IEEE Trans Med Imaging 2019; 38(4): 945–954

    PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Science and Technology Major Project of the Ministry of Science and Technology of China (No. 2017YFC0110903), Microsoft Research under the eHealth program, the National Natural Science Foundation of China (No. 81771910), the Beijing Natural Science Foundation in China (No. 4152033), the Technology and Innovation Commission of Shenzhen in China (No. shenfagai2016-627), the Beijing Young Talent Project in China, the Fundamental Research Funds for the Central Universities of China (No. SKLSDE-2017ZX-08) from the State Key Laboratory of Software Development Environment in Beihang University in China, and the 111 Project in China (No. B13003).

Author information

Author notes

  1. Equal co-authorship, ordered alphabetically.

Authors and Affiliations

  1. School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China

    Shujian Deng, Xin Zhang, Wen Yan, Yubo Fan & Yan Xu

  2. Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education and State Key Laboratory of Software Development Environment, Beihang University, Beijing, 100191, China

    Shujian Deng, Xin Zhang, Wen Yan, Yubo Fan & Yan Xu

  3. Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China

    Shujian Deng, Xin Zhang, Wen Yan, Yubo Fan & Yan Xu

  4. Microsoft Research Asia, Beijing, 100080, China

    Eric I-Chao Chang & Yan Xu

  5. Department of Pathology, School of Medicine, Zhejiang University, Hangzhou, 310007, China

    Maode Lai

Authors
  1. Shujian Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric I-Chao Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yubo Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Maode Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yan Xu.

Ethics declarations

Shujian Deng, Xin Zhang, Wen Yan, Eric I-Chao Chang, Yubo Fan, Maode Lai, and Yan Xu declare that they have no conflicts of interest. This manuscript is a review article and does not involve a research protocol requiring approval by the relevant institutional review board or ethics committee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, S., Zhang, X., Yan, W. et al. Deep learning in digital pathology image analysis: a survey. Front. Med. 14, 470–487 (2020). https://doi.org/10.1007/s11684-020-0782-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11684-020-0782-9

Keywords

Search

Navigation