Nuclei Detection in Cytological Images Using Convolutional Neural Network and Ellipse Fitting Algorithm

  • Marek KowalEmail author
  • Michał Żejmo
  • Józef Korbicz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10842)


Morphometric analysis of nuclei play an essential role in cytological diagnostics. Cytological samples contain hundreds or thousands of nuclei that need to be examined for cancer. The process is tedious and time-consuming but can be automated. Unfortunately, segmentation of cytological samples is very challenging due to the complexity of cellular structures. To deal with this problem, we are proposing an approach, which combines convolutional neural network and ellipse fitting algorithm to segment nuclei in cytological images of breast cancer. Images are preprocessed by the colour deconvolution procedure to extract hematoxylin-stained objects (nuclei). Next, convolutional neural network is performing semantic segmentation of preprocessed image to extract nuclei silhouettes. To find the exact location of nuclei and to separate touching and overlapping nuclei, we approximate them using ellipses of various sizes and orientations. They are fitted using the Bayesian object recognition approach. The accuracy of the proposed approach is evaluated with the help of reference nuclei segmented manually. Tests carried out on breast cancer images have shown that the proposed method can accurately segment elliptic-shaped objects.


Deep learning Convolutional neural network Ellipse fitting Bayesian object recognition Nuclei detection Breast cancer 



The research was supported by National Science Centre, Poland (2015/17/B/ST7/03704).


  1. 1.
    Spanhol, F.A., Oliveira, S.L.E., Petitjean, C., Heutte, L.: Breast cancer histopathological image classification using convolutional neural networks. In: Proceedings of International Conference on Neural Networks (IJCNN 2016), Vancouver, Canada (2016)Google Scholar
  2. 2.
    Bembenik, R., Jóźwicki, W., Protaziuk, G.: Methods for mining co-location patterns with extended spatial objects. Int. J. Appl. Math. Comp. Sci. 27(4), 681–695 (2017)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Qi, J.: Dense nuclei segmentation based on graph cut and convexity–concavity analysis. J. Microsc. 253(1), 42–53 (2014)CrossRefGoogle Scholar
  4. 4.
    Kłeczek, P., Dyduch, G., Jaworek-Korjakowska, J., Tadeusiewicz, R.: Automated epidermis segmentation in histopathological images of human skin stained with hematoxylin and eosin. In: Proceedings of SPIE Medical Imaging, vol. 10140, pp. 10140–10140–19 (2017)Google Scholar
  5. 5.
    Nurzynska, K., Mikhalkin, A., Piórkowski, A.: CAS: cell annotation software - research on neuronal tissue has never been so transparent. Neuroinformatics 15, 365–382 (2017)CrossRefGoogle Scholar
  6. 6.
    Kowal, M., Filipczuk, P.: Nuclei segmentation for computer-aided diagnosis of breast cancer. Int. J. Appl. Math. Comp. Sci. 24(1), 19–31 (2014)CrossRefGoogle Scholar
  7. 7.
    Chu, J.L., Krzyżak, A.: The recognition of partially occluded objects with support vector machines, convolutional neural networks and deep belief networks. J. Artif. Intell. Soft Comput. Res. 4(1), 5–19 (2014)CrossRefGoogle Scholar
  8. 8.
    Surya, S., Babu, R.V.: TraCount: a deep convolutional neural network for highly overlapping vehicle counting. In: Proceedings of 10th Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 2016, pp. 46:1–46:6, New York, NY, USA (2016)Google Scholar
  9. 9.
    Hu, R.L., Karnowski, J., Fadely, R., Pommier, J.P.: Image segmentation to distinguish between overlapping human chromosomes. In: Proceedings of 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA (2017)Google Scholar
  10. 10.
    Descombes, X.: Multiple objects detection in biological images using a marked point process framework. Methods 115(Supplement C), 2–8 (2017). Image Processing for BiologistsCrossRefGoogle Scholar
  11. 11.
    Ruifrok, A.C., Johnston, D.A.: Quantification of histochemical staining by color deconvolution. Anal. Quant. Cytol. Histol. 23(4), 291–299 (2001)Google Scholar
  12. 12.
    LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2004, vol. 2, pp. II–97. IEEE (2004)Google Scholar
  13. 13.
    van Lieshout, M.N.M.: Markov point processes and their applications in high-level imaging. Bull. Int. Stat. Inst. 56, 559–576 (1995)zbMATHGoogle Scholar
  14. 14.
    Kowal, M., Korbicz, J.: Marked point process for nuclei detection in breast cancer microscopic images. In: Augustyniak, P., Maniewski, R., Tadeusiewicz, R. (eds.) PCBBE 2017. AISC, vol. 647, pp. 230–241. Springer, Cham (2018). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Institute of Control and Computation EngineeringUniversity of Zielona GóraZielona GóraPoland

Personalised recommendations