World Wide Web

, Volume 21, Issue 6, pp 1721–1743 | Cite as

Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks

  • Xipeng Pan
  • Dengxian Yang
  • Lingqiao Li
  • Zhenbing Liu
  • Huihua YangEmail author
  • Zhiwei Cao
  • Yubei He
  • Zhen Ma
  • Yiyi Chen
Part of the following topical collections:
  1. Special Issue on Deep Mining Big Social Data


Automated nucleus/cell detection is usually considered as the basis and a critical prerequisite step of computer assisted pathology and microscopy image analysis. However, due to the enormous variability (cell types, stains and different microscopes) and data complexity (cell overlapping, inhomogeneous intensities, background clutters and image artifacts), robust and accurate nucleus/cell detection is usually a difficult problem. To address this issue, we propose a novel multi-scale fully convolutional neural networks approach for regression of a density map to robustly detect the nuclei of pathology and microscopy images. The procedure can be divided into three main stages. Initially, instead of working on the simple dot label space, regression on the proposed structured proximity space for patches is performed so that centers of image patches are explicitly forced to produce larger values than their adjacent areas. Then, several multi-scale fully convolutional regression networks are developed for this task; this will enlarge the receptive field and not only can detect the single, small size cells, but also benefit to detecting cells with big size and overlapping states. In this stage, we copy the full feature maps from the contracting path and merge with the feature maps of the expansive path. This operation will make full use of shallow and deep semantic information of the networks. The networks do not have any fully connected layers; this strategy allows the seamless probability map prediction of arbitrarily large images. At the same time, data augmentations (e.g., small range shift, zoom and randomly flip) are carefully used to enhance the robustness of detection. Finally, morphological operations and suitable filters are employed and some prior information is introduced to find the centers of the cells more robustly. Our method achieves about 99.25% detection precision and the F1-measure is 0.9924 on fluorescence microscopy cell images; about 85.90% detection precision and the F1-measure is 0.9020 on Lymphocyte cell images and about 78.41% detection precision and the F1-measure is 0.8440 on breast histopathological images. This result leads to a promising detection performance that equates and sometimes exceeds the recently published leading detection approaches with the same benchmark datasets.


Cell detection Fully convolutional neural networks Pathology and microscopy images Multi-scale 



The authors would like to thank Lehmussola et al. [23], Dr. Andrew Janowczyk et al. [2], and Dr. Zhang et al. [52] for publishing the datasets. We are grateful for helpful comments from the anonymous reviewers and the associate editor. This research was supported in part by the National Natural Science Foundation of China (Grant Nos. 21365008 and 61562013), and Natural Science Foundation of Guangxi Province (No. 2017GXNSFDA198025).


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., SüSstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell. 34(11), 2274 (2012)CrossRefGoogle Scholar
  2. 2.
    Andrew, J., Anant, M.: Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. Journal of Pathology Informatics. 7(1), 29 (2016)CrossRefGoogle Scholar
  3. 3.
    Arteta, C., Lempitsky, V., Noble, J. A., Zisserman, A.: Learning to detect cells using non-overlapping extremal regions. In: Medical Image Computing & Computer-assisted Intervention: Miccai International Conference on Medical Image Computing & Computer-assisted Intervention, 15, 348–356 (2012)Google Scholar
  4. 4.
    Arteta, C., Lempitsky, V., Noble, J.A., Zisserman, A.: Learning to detect cells using non-overlapping extremal regions. In: International Conference on Medical Image Computing and Computer Assisted Intervention (Lecture Notes in Computer Science). MICCAI, 15, pp. 348–356 (2012)Google Scholar
  5. 5.
    Arteta, C., Lempitsky, V., Noble, J. A., & Zisserman, A.: Interactive object counting. In: Proceedings of the European Conference on Computer Vision (ECCV), 8691, 504–518 (2014)Google Scholar
  6. 6.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Mach Intell. 26(9), 1124 (2004)CrossRefGoogle Scholar
  7. 7.
    Cai, Z., Fan, Q., Feris, R. S., Vasconcelos, N.: A unified multi-scale deep convolutional neural network for fast object detection. In: Leibe B., Matas J., Sebe N., Welling M. (eds.) European Conference on Computer Vision. pp.354–370. Springer, Cham (2016)CrossRefGoogle Scholar
  8. 8.
    Cireşan, D. C., Giusti, A., Gambardella, L. M., Schmidhuber, J.: Mitosis detection in breast cancer histology images with deep neural networks. In: International Conference on Medical Image Computing & Computer-assisted Intervention, 16, 411–418 (2013)Google Scholar
  9. 9.
    Cruz-Roa A.A., Arevalo Ovalle J.E., Madabhushi A., González Osorio F.A.: A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. In: Mori K., Sakuma I., Sato Y., Barillot C., Navab N. (eds.) International Conference on Medical Image Computing and Computer-assisted Intervention (MICCAI), pp. 403–410. Springer, Berlin (2013)CrossRefGoogle Scholar
  10. 10.
    Dan, C.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. Adv. Neural Inf. Proces. Syst. 25, 2852–2860 (2012)Google Scholar
  11. 11.
    Dong, B., Shao, L., Costa, M. D., Bandmann, O., Frangi, A. F.: Deep learning for automatic cell detection in wide-field microscopy zebrafish images. In: Proceedings of IEEE 12th International Symposium on Biomedical Imaging (ISBI), pp. 772–776. IEEE New York (2015)Google Scholar
  12. 12.
    Dundar, M.M., Badve, S., Bilgin, G., Raykar, V., Jain, R., Sertel, O., et al.: Computerized classification of intraductal breast lesions using histopathological images. IEEE Trans. Biomed. Eng. 58(7), 1977–1984 (2011)CrossRefGoogle Scholar
  13. 13.
    Fatakdawala, H., Xu, J., Basavanhally, A., Bhanot, G., Ganesan, S., Feldman, M., et al.: Expectation-maximization-driven geodesic active contour with overlap resolution (emagacor): application to lymphocyte segmentation on breast cancer histopathology. IEEE Trans. Biomed. Eng. 57(7), 1676–1689 (2010)CrossRefGoogle Scholar
  14. 14.
    Fiaschi, L., Koethe, U., Nair, R., Hamprecht, F. A.: Learning to count with regression forest and structured labels. In: Proceedings of 21st International Conference on Pattern Recognition (ICPR), pp. 2685–2688. IEEE (2012)Google Scholar
  15. 15.
    Foran, D.J., Yang, L., Chen, W., Hu, J., Goodell, L.A., Reiss, M., et al.: Imageminer: a software system for comparative analysis of tissue microarrays using content-based image retrieval, high-performance computing, and grid technology. J. Am. Med. Inform. Assoc. 18(4), 403–415 (2011)CrossRefGoogle Scholar
  16. 16.
    García-Gojo, M.: State of the art and trends for digital pathology. Stud Health Technol Inform. 179, 15–28 (2012)Google Scholar
  17. 17.
    Giusti, A., Dan, C.C., Masci, J., Gambardella, L.M., Schmidhuber, J.: Fast image scanning with deep max-pooling convolutional. Neural Netw. 4034–4038 (2013)Google Scholar
  18. 18.
    Guan, B.: Cell segmentation: 50 years down the road. IEEE Signal Process. Mag. 29(5), 140–145 (2012)CrossRefGoogle Scholar
  19. 19.
    Hu, R., Zhu, X., Cheng, D., et al.: Graph self-representation method for unsupervised feature selection. Neurocomputing. 220, 130–137 (2017)CrossRefGoogle Scholar
  20. 20.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. International Conference on International Conference on Machine Learning. 448–456 (2015)Google Scholar
  21. 21.
    Khoshdeli, M., Cong R., Parvin, B.: Detection of nuclei in H&E stained sections using convolutional neural networks. In: 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 105–108. IEEE (2017).
  22. 22.
    Kuse, M., Wang, Y.F., Kalasannavar, V., Khan, M., Rajpoot, N.: Local isotropic phase symmetry measure for detection of beta cells and lymphocytes. Journal of Pathology Informatics. 2(2), S2 (2011)CrossRefGoogle Scholar
  23. 23.
    Lehmussola, A., Ruusuvuori, P., Selinummi, J., Huttunen, H., Yli-Harja, O.: Computational framework for simulating fluorescence microscope images with cell populations. IEEE Trans. Med. Imaging. 26(7), 1010–1016 (2007)CrossRefGoogle Scholar
  24. 24.
    Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Advances in Neural Information Processing Systems (NIPS). 43, 1324–1332 (2010)Google Scholar
  25. 25.
    Li, X., & Plataniotis, K. N.: Color model comparative analysis for breast cancer diagnosis using h and e stained images. In: SPIE Medical Imaging International Society for Optics and Photonics, 9420, 94200L–94200L-6 (2015)Google Scholar
  26. 26.
    Liu, F., Yang, L.: A novel cell detection method using deep convolutional neural network and maximum-weight independent set. In: Lu L., Zheng Y., Carneiro G., Yang L. (eds.) Deep Learning and Convolutional Neural Networks for Medical Image Computing, Advances in Computer Vision and Pattern Recognition, pp. 349-357. Springer, Cham (2015)Google Scholar
  27. 27.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence. 39(4), 640–651 (2014)Google Scholar
  28. 28.
    López, C., Lejeune, M., Bosch, R., Korzynska, A., Garcíarojo, M., Salvadó, M.T., et al.: Digital image analysis in breast cancer: an example of an automated methodology and the effects of image compression. Stud Health Technol Inform. 179, 155–171 (2012)Google Scholar
  29. 29.
    Pan, X., Li, L., Yang, H., et al.: Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing. 229, 88–99 (2017)CrossRefGoogle Scholar
  30. 30.
    Parvin, B., Yang, Q., Han, J., Chang, H., Rydberg, B., Barcelloshoff, M.H.: Iterative voting for inference of structural saliency and characterization of subcellular events. IEEE transactions on image processing a publication of the IEEE signal processing. Society. 16(3), 615–623 (2007)Google Scholar
  31. 31.
    Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M. H.: Single Image Dehazing via Multi-scale Convolutional Neural Networks. In: European Conference on Computer Vision. pp. 154–169. Springer (2016)Google Scholar
  32. 32.
    Ronneberger, O., Fischer, P., & Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 9351, 234–241 (2015)Google Scholar
  33. 33.
    Saxe, A. M., Mcclelland, J. L., Ganguli, S.: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120. (2014)Google Scholar
  34. 34.
    Sirinukunwattana, K., Shan, E. A. R., Tsang, Y. W., Snead, D., Cree, I., Rajpoot, N.: A Spatially Constrained Deep Learning Framework for Detection of Epithelial Tumor Nuclei in Cancer Histology Images. In: Wu G., Coupé P., Zhan Y., Munsell B., Rueckert D. (eds.) Patch-Based Techniques in Medical Imaging. Lecture Notes in Computer Science, vol. 9467, pp. 154–162. Springer, Cham (2015)CrossRefGoogle Scholar
  35. 35.
    Sirinukunwattana, K., Raza, S., Tsang, Y.W., Snead, D., Cree, I., Rajpoot, N.: Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans. Med. Imaging. 35(5), 1196–1206 (2016)CrossRefGoogle Scholar
  36. 36.
    Sommer, C., Hoefler, R., Samwer, M., et al.: A deep learning and novelty detection framework for rapid phenotyping in high-content screening. Mol. Biol. Cell. (2017).
  37. 37.
    Song, Y., Zhang, L., Chen, S., Ni, D., Lei, B., Wang, T.: Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning. IEEE Trans. Biomed. Eng. 62(10), 2421–2433 (2015)CrossRefGoogle Scholar
  38. 38.
    Song TH., Sanchez V., EIDaly H., Rajpoot N.: Simultaneous cell detection and classification with an asymmetric deep autoencoder in bone marrow histology images. In: Valdés Hernández M. and González-Castro V. (eds.) Medical Image Understanding and Analysis. MIUA 2017. Communications in Computer and Information Science, vol 723. pp. 829–838. Springer, Cham (2017)Google Scholar
  39. 39.
    Song, T., Sanchez, V., Eidaly, H., Rajpoot, N.: Hybrid deep autoencoder with Curvature Gaussian for detection of various types of cells in bone marrow trephine biopsy images. In: 2017 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1040–1043. IEEE (2017)Google Scholar
  40. 40.
    Su, H., Yin, Z., Kanade, T., & Huh, S.: Phase contrast image restoration via dictionary representation of diffraction patterns. In: International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, 615–622 (2012)Google Scholar
  41. 41.
    Su, H., Xing, F., Kong, X., Xie, Y., Zhang, S., Yang, L.: Robust cell detection and segmentation in histopathological images using sparse reconstruction and stacked denoising autoencoders. In: Navab N., Hornegger J., Wells W., Frangi A. (eds.) Medical Image Computing and Computer-Assisted Intervention (MICCAI), Lecture Notes in Computer Science, vol. 9351, pp. 383–390. Springer, Cham (2015)Google Scholar
  42. 42.
    Wang, H., Cruz-Roa, A., Basavanhally, A., et al.: Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. Journal of Medical Imaging, 1(3), 034003–1-8 (2014)CrossRefGoogle Scholar
  43. 43.
    Wei, S., Zhou, M., Yang, F., Yang, C., Tian, J.: Multi-scale Convolutional Neural Networks for Lung Nodule Classification. In: Ourselin S., Alexander D., Westin CF., Cardoso M. (eds.) Information Processing in Medical Imaging (IPMI), vol. 9123, pp. 588–599. Springer, Cham (2015)Google Scholar
  44. 44.
    Xie, Y., Kong, X., Xing, F., Liu, F., Su, H., Yang, L.: Deep voting: a robust approach toward nucleus localization in microscopy images. In: Navab N., Hornegger J., Wells W., Frangi A. (eds.) Medical Image Computing and Computer-Assisted Intervention (MICCAI). Lecture Notes in Computer Science, vol 9351. pp. 374–382. Springer, Cham (2015)Google Scholar
  45. 45.
    Xie, Y., Xing, F., Kong, X., Su, H., Yang, L.: Beyond Classification: Structured Regression for Robust Cell Detection Using Convolutional Neural Network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, vol 9351. pp. 358–365. Springer, (2015)Google Scholar
  46. 46.
    Xie, W., Noble, J. A., Zisserman, A.: Microscopy cell counting and detection with fully convolutional regression networks. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, pp. 1–10. Taylor & Francis, Oxfordshire (2016)Google Scholar
  47. 47.
    Xing, F., Yang, L.: Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review. IEEE Rev. Biomed. Eng. 9, 234 (2016)CrossRefGoogle Scholar
  48. 48.
    Xing, F., Xie, Y., Yang, L.: An automatic learning-based framework for robust nucleus segmentation. IEEE Trans. Med. Imaging. 35(2), 550–566 (2015)CrossRefGoogle Scholar
  49. 49.
    Xu, J., Xiang, L., Liu, Q., Gilmore, H., Wu, J., Tang, J., et al.: Stacked sparse autoencoder (ssae) for nuclei detection on breast cancer histopathology images. IEEE Trans. Med. Imaging. 35(1), 119 (2016)CrossRefGoogle Scholar
  50. 50.
    Yellin, F., Haeffele, B. D., Vidal, R.: Blood cell detection and counting in holographic lens-free imaging by convolutional sparse dictionary learning and coding. IEEE International Symposium on Biomedical Imaging, 650–653 (2017)Google Scholar
  51. 51.
    Zhang, S., Metaxas, D.: Large-scale medical image analytics: recent methodologies, applications and future directions. Med. Image Anal. 33, 98–101 (2016)CrossRefGoogle Scholar
  52. 52.
    Zhang, X., Liu, W., Dundar, M., Badve, S., Zhang, S.: Towards large-scale histopathological image analysis: hashing-based image retrieval. IEEE Trans. Med. Imaging. 34(2), 496–506 (2015)CrossRefGoogle Scholar
  53. 53.
    Zhang, X., Xing, F., Su, H., Yang, L., Zhang, S.: High-throughput histopathological image analysis via robust cell segmentation and hashing. Med. Image Anal. 26(1), 306–315 (2015)CrossRefGoogle Scholar
  54. 54.
    Zhu, X., Zhang, L., Huang, Z.: A sparse embedding and least variance encoding approach to hashing. IEEE Trans. Image Process. 23(9), 3737–3750 (2014)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Zhu, X., Suk, H.I., Wang, L., et al.: A novel relational regularization feature selection method for joint regression and classification in AD diagnosis. Med. Image Anal. 75(6), 570–577 (2015)Google Scholar
  56. 56.
    Zhu, X., Li, X., Zhang, S.: Block-row sparse multiview multilabel learning for image classification. IEEE Transactions on Cybernetics. 46(2), 450–461 (2016)CrossRefGoogle Scholar
  57. 57.
    Zhu, X., Suk, H.I., Lee, S.W., et al.: Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification. IEEE Trans. Biomed. Eng. 63(3), 607–618 (2016)CrossRefGoogle Scholar
  58. 58.
    Zhu, X., Li, X., Zhang, S., et al.: Robust joint graph sparse coding for unsupervised spectral feature selection. IEEE Transactions on Neural Networks & Learning Systems. 28(6), 1263–1275 (2017)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Xipeng Pan
    • 1
  • Dengxian Yang
    • 2
  • Lingqiao Li
    • 1
    • 3
  • Zhenbing Liu
    • 3
  • Huihua Yang
    • 1
    • 3
    Email author
  • Zhiwei Cao
    • 1
  • Yubei He
    • 4
  • Zhen Ma
    • 1
  • Yiyi Chen
    • 1
  1. 1.School of AutomationBeijing University of Posts and TelecommunicationsBeijingChina
  2. 2.College of Arts and SciencesUniversity of Washington – SeattleWashingtonUSA
  3. 3.School of Computer Science and Information SecurityGuilin University of Electronic TechnologyGuilinChina
  4. 4.School of Computing and Information SystemsUniversity of MelbourneMelbourneAustralia

Personalised recommendations