Abstract
Nuclei instance segmentation is an integral step in digital pathology workflow as it is a prerequisite for most downstream tasks such as patient survival analysis, precision medicine, and cancer prognosis. There exist many challenges such as quality of labeled data, staining variation among tissue slides, high variation among multi-organ & multi-center digital slides and overlapping nuclei that are difficult to separate. Therefore, it is important to have an automatic and robust nuclei instance segmentation model that saves the time of pathologists by delineating accurate nuclei instances. To this end, we develop a nuclei instance segmentation pipeline that estimates distance transform and nuclear masks using an encoder–decoder-based CNN model. These estimated distance transform and nuclear masks are then utilized to delineate accurate nuclei boundaries from overlapping nuclei. We demonstrate that our proposed NC-Net model is lightweight and produces state-of-the-art results on the three recently published largest nuclei instance segmentation datasets to date. Additionally, our proposed NC-Net model is faster and utilizes a fewer number of parameters for learning as compared to other top-performing nuclei instance segmentation models. The purpose of developing a lightweight and state-of-the-art model is to provide capacity building to digital pathology workflows by reducing inference times and delineating accurate nuclear instances. The implementation details and the trained models are made available at this https://github.com/nauyan/NC-Net.
Similar content being viewed by others
Data availability
Data sharing is not applicable, as no new data are generated.
References
Cancer. https://www.who.int/news-room/fact-sheets/detail/cancer. Accessed 2020-11-08
Chow AY (2010) Cell cycle control by oncogenes and tumor suppressors: driving the transformation of normal cells into cancerous cells. Nat Educ 3(9):7
Fischer AH, Jacobson KA, Rose J, Zeller R (2008) Hematoxylin and eosin staining of tissue and cell sections. Cold Spring Harbor Protocols 2008(5):4986
Elmore JG, Longton GM, Carney PA, Geller BM, Onega T, Tosteson AN, Nelson HD, Pepe MS, Allison KH, Schnitt SJ et al (2015) Diagnostic concordance among pathologists interpreting breast biopsy specimens. Jama 313(11):1122–1132
Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A (2019) Artificial intelligence in digital pathology-new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 16(11):703–715
Madabhushi A, Lee G (2016) Image analysis and machine learning in digital pathology: challenges and opportunities. Med Image Anal 33:170–175
Alsubaie N, Sirinukunwattana K, Raza SEA, Snead D, Rajpoot N ( 2018) A bottom-up approach for tumour differentiation in whole slide images of lung adenocarcinoma. In: Medical imaging 2018: digital pathology, vol 10581, p 105810. International Society for Optics and Photonics
Lu C, Romo-Bucheli D, Wang X, Janowczyk A, Ganesan S, Gilmore H, Rimm D, Madabhushi A (2018) Nuclear shape and orientation features from h &e images predict survival in early-stage estrogen receptor-positive breast cancers. Lab Invest 98(11):1438–1448
Lu C, Romo-Bucheli D, Wang X, Janowczyk A, Ganesan S, Gilmore H, Rimm D, Madabhushi A (2018) Nuclear shape and orientation features from h &e images predict survival in early-stage estrogen receptor-positive breast cancers. Lab Invest 98(11):1438–1448
Colling R, Pitman H, Oien K, Rajpoot N, Macklin P, Histopathology Working Group CPA, Bachtiar V, Booth R, Bryant A, Bull J, et al (2019) Artificial intelligence in digital pathology: a roadmap to routine use in clinical practice. J Pathol 249( 2), 143– 150
Jimenez-del-Toro O, Otálora S, Andersson M, Eurén K, Hedlund M, Rousson M, Müller H, Atzori M ( 2017) Analysis of histopathology images: from traditional machine learning to deep learning. In: Biomedical texture analysis, pp. 281– 314. Academic Press
Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A (2019) Artificial intelligence in digital pathology-new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 16(11):703–715
Gurcan MN, Boucheron LE, Can A, Madabhushi A, Rajpoot NM, Yener B (2009) Histopathological image analysis: a review. IEEE Rev Biomed Eng 2:147–171
Yang X, Li H, Zhou X (2006) Nuclei segmentation using marker-controlled watershed, tracking using mean-shift, and kalman filter in time-lapse microscopy. IEEE Trans Circuits Syst I Regular Pap 53(11):2405–2414
Veta M, Van Diest PJ, Kornegoor R, Huisman A, Viergever MA, Pluim JP (2013) Automatic nuclei segmentation in h &e stained breast cancer histopathology images. PloS One 8(7):70221
Ali S, Madabhushi A (2012) An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery. IEEE Trans Med Imag 31(7):1448–1460
Wienert S, Heim D, Saeger K, Stenzinger A, Beil M, Hufnagl P, Dietel M, Denkert C, Klauschen F (2012) Detection and segmentation of cell nuclei in virtual microscopy images: a minimum-model approach. Sci Rep 2(1):1–7
LaTorre A, Alonso-Nanclares L, Muelas S, Peña J, DeFelipe J (2013) Segmentation of neuronal nuclei based on clump splitting and a two-step binarization of images. Exp Syst Appl 40(16):6521–6530
Kwak JT, Hewitt SM, Xu S, Pinto PA, Wood BJ ( 2015) Nucleus detection using gradient orientation information and linear least squares regression. In: Medical imaging 2015: digital pathology, vol 9420, pp 152– 159. SPIE
Liao M, Zhao Y-q, Li X-h, Dai P-s, Xu X-w, Zhang J-k, Zou B-j ( 2016) Automatic segmentation for cell images based on bottleneck detection and ellipse fitting. Neurocomputing 173:615–622
LeCun Y, Bengio Y, Hinton G et al (2015) Deep learning. Nature 521(7553):436–444 (Google Scholar Google Scholar Cross Ref Cross Ref)
Wu Y, Ji X, Ji W, Tian Y, Zhou H (2020) Casr: a context-aware residual network for single-image super-resolution. Neural Comput Appl 32(18):14533–14548
Dogar GM, Shahzad M, Fraz MM (2023) Attention augmented distance regression and classification network for nuclei instance segmentation and type classification in histology images. Biomed Signal Process Control 79:104199
Nasir ES, Perviaz A, Fraz MM (2022) Nuclei and glands instance segmentation in histology images: a narrative review. arXiv preprint arXiv:2208.12460
Fraz M, Khurram SA, Graham S, Shaban M, Hassan M, Loya A, Rajpoot NM (2020) Fabnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer. Neural Comput Appl 32(14):9915–9928
Shaban M, Khurram SA, Fraz MM, Alsubaie N, Masood I, Mushtaq S, Hassan M, Loya A, Rajpoot NM (2019) A novel digital score for abundance of tumour infiltrating lymphocytes predicts disease free survival in oral squamous cell carcinoma. Sci Rep 9(1):1–13
Rasool A, Fraz MM, Javed S( 2021) Multiscale unified network for simultaneous segmentation of nerves and micro-vessels in histology images. In: 2021 International conference on digital futures and transformative technologies (ICoDT2), pp. 1– 6. IEEE
Bashir RS, Mahmood H, Shaban M, Raza SEA, Fraz MM, Khurram SA, Rajpoot NM ( 2020) Automated grade classification of oral epithelial dysplasia using morphometric analysis of histology images. In: Medical imaging 2020: digital pathology, vol 11320, pp 245–250. SPIE
Javed S, Mahmood A, Fraz MM, Koohbanani NA, Benes K, Tsang Y-W, Hewitt K, Epstein D, Snead D, Rajpoot N (2020) Cellular community detection for tissue phenotyping in colorectal cancer histology images. Med Image Anal 63:101696
Dogar GM, Fraz MM, Javed S( 2021) Feature attention network for simultaneous nuclei instance segmentation and classification in histology images. In: 2021 International conference on digital futures and transformative technologies (ICoDT2), pp 1–6. IEEE
Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28
Lin T-Y, Goyal P, Girshick R, He K, Dollár P ( 2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988
Long J, Shelhamer E, Darrell T ( 2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440
Ronneberger O, Fischer P, Brox T ( 2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234– 241. Springer
Raza SEA, Cheung L, Shaban M, Graham S, Epstein D, Pelengaris S, Khan M, Rajpoot NM (2019) Micro-net: A unified model for segmentation of various objects in microscopy images. Med Image Anal 52:160–173
Graham S, Rajpoot NM ( 2018) Sams-net: Stain-aware multi-scale network for instance-based nuclei segmentation in histology images. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 590– 594. IEEE
Chen H, Qi X, Yu L, Heng P-A ( 2016) Dcan: deep contour-aware networks for accurate gland segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2487–2496
Cui Y, Zhang G, Liu Z, Xiong Z, Hu J (2019) A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images. Med Biol Eng Comput 57(9):2027–2043
Kumar N, Verma R, Sharma S, Bhargava S, Vahadane A, Sethi A (2017) A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans Med Imag 36(7):1550–1560
Khoshdeli M, Parvin B (2018) Deep learning models delineates multiple nuclear phenotypes in h & e stained histology sections. arXiv preprint arXiv:1802.04427
Zhou Y, Onder OF, Dou Q, Tsougenis E, Chen H, Heng P-A ( 2019) Cia-net: Robust nuclei instance segmentation with contour-aware information aggregation. In: International conference on information processing in medical imaging, pp 682– 693. Springer
Vu QD, Graham S, Kurc T, To MNN, Shaban M, Qaiser T, Koohbanani NA, Khurram SA, Kalpathy-Cramer J, Zhao T et al (2019) Methods for segmentation and classification of digital microscopy tissue images. Front Bioeng Biotechnol, 53
Graham S, Vu QD, Raza SEA, Azam A, Tsang YW, Kwak JT, Rajpoot N (2019) Hover-net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal 58:101563
Gamper J, Koohbanani NA, Benes K, Graham S, Jahanifar M, Khurram SA, Azam A, Hewitt K, Rajpoot N (2020) Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778
Graham S, Jahanifar M, Azam A, Nimir M, Tsang Y-W, Dodd K, Hero E, Sahota H, Tank A, Benes K, et al (2021) Lizard: A large-scale dataset for colonic nuclear instance segmentation and classification. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 684– 693
Alemi Koohbanani N, Jahanifar M, Gooya A, Rajpoot N ( 2019) Nuclear instance segmentation using a proposal-free spatially aware deep learning framework. In: International conference on medical image computing and computer-assisted intervention, pp 622–630. Springer
Chen S, Ding C, Tao D (2020) Boundary-assisted region proposal networks for nucleus segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 279– 288. Springer
Zhao B, Chen X, Li Z, Yu Z, Yao S, Yan L, Wang Y, Liu Z, Liang C, Han C (2020) Triple u-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation. Med Image Anal 65:101786
Schmidt U, Weigert M, Broaddus C, Myers G ( 2018) Cell detection with star-convex polygons. In: International conference on medical image computing and computer-assisted intervention, pp 265– 273 . Springer
Chen, S., Ding, C., Liu, M., Tao, D.: Cpp-net: Context-aware polygon proposal network for nucleus segmentation. arXiv preprint arXiv:2102.06867 (2021)
Tan M, Le Q( 2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning, pp 6105–6114. PMLR
Roy AG, Navab N, Wachinger C (2018) Recalibrating fully convolutional networks with spatial and channel squeeze and excitation blocks. IEEE Trans Med Imag 38(2):540–549
Vincent L, Soille P (1991) Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans Pattern Anal Mach Intell 13(06):583–598
Kirillov A, He K, Girshick R, Rother C, Dollár P ( 2019) Panoptic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9404– 9413
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al (2019) Pytorch: An imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 32
Bradski G, Kaehler A (2000) Opencv. Dr. Dobb’s journal of software tools 3:2
Van der Walt S, Schönberger JL, Nunez-Iglesias J, Boulogne F, Warner JD, Yager N, Gouillart E, Yu T (2014) scikit-image: image processing in python. Peer J 2:453
Buslaev A, Iglovikov VI, Khvedchenya E, Parinov A, Druzhinin M, Kalinin AA (2020) Albumentations: fast and flexible image augmentations. Information 11(2):125
Yakubovskiy, P(2020) Segmentation Models Pytorch. GitHub https://github.com/qubvel
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L( 2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255 . IEEE
Bock S, Goppold J, Weiß M(2018) An improvement of the convergence proof of the adam-optimizer. arXiv preprint arXiv:1804.10587
Carpenter AE, Jones TR, Lamprecht MR, Clarke C, Kang IH, Friman O, Guertin DA, Chang JH, Lindquist RA, Moffat J et al (2006) Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol 7(10):1–11
Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481–2495
Chen RJ, Lu MY, Wang J, Williamson DF, Rodig SJ, Lindeman NI, Mahmood F (2020) Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans Medi Imag
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflit of interest
The authors declare no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rashid, S.N., Fraz, M.M. Nuclei probability and centroid map network for nuclei instance segmentation in histology images. Neural Comput & Applic 35, 15447–15460 (2023). https://doi.org/10.1007/s00521-023-08503-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-023-08503-2