Skip to main content

A Holistically-Nested U-Net: Surgical Instrument Segmentation Based on Convolutional Neural Network

Journal of Digital Imaging volume 33, pages 341–347 (2020)Cite this article

Abstract

Surgical instrument segmentation is an essential task in the domain of computer-assisted surgical system. It is critical to increase the context-awareness of surgeons during the operation. We propose a new model based on the U-Net architecture for surgical instrument segmentation, which aggregates multi-scale feature maps and has cascaded dilated convolution layers. The model adopts dense upsampling convolution instead of deconvolution for upsampling. We set the side loss function on each side-output layer. The loss function includes an output loss function and all side loss functions to supervise the training of each layer. To validate our model, we compare our proposed model with advanced architecture U-Net in the dataset consisting of laparoscopy images from multiple surgical operations. Experiment results demonstrate that our model achieves good performance.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Bouget D, Allan M, Stoyanov D, Jannin P: Vision-based and marker-less surgical tool detection and tracking: A review of the literature. Med Image Anal 35:633–654, 2017. https://doi.org/10.1016/j.media.2016.09.003

    Article  PubMed  Google Scholar 

  2. Baumhauer M, Feuerstein M, Meinzer H-P, Rassweiler J: Navigation in endoscopic soft tissue surgery: Perspectives and limitations. J Endourol 22:751–766, 2008. https://doi.org/10.1089/end.2007.9827

    Article  PubMed  Google Scholar 

  3. Lalys F, Jannin P: Surgical process modelling: A review. Int J Comput Assist Radiol Surg 9:495–511, 2014. https://doi.org/10.1007/s11548-013-0940-5

    Article  PubMed  Google Scholar 

  4. Doignon C, Nageotte F, De Mathelin M: Detection of grey regions in color images: Application to the segmentation of a surgical instrument in robotized laparoscopy, in: 2004 IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IEEE Cat. No.04CH37566), IEEE, n.d.: pp. 3394–3399. https://doi.org/10.1109/IROS.2004.1389941

  5. Bouget D, Benenson R, Omran M, Riffaud L, Schiele B, Jannin P: Detecting surgical tools by modelling local appearance and global shape. IEEE Trans Med Imaging 34:2603–2617, 2015. https://doi.org/10.1109/TMI.2015.2450831

    Article  PubMed  Google Scholar 

  6. Allan M, Ourselin S, Thompson S, Hawkes DJ, Kelly J, Stoyanov D: Toward detection and localization of instruments in minimally invasive surgery. IEEE Trans Biomed Eng 60:1050–1058, 2013. https://doi.org/10.1109/TBME.2012.2229278

    Article  PubMed  Google Scholar 

  7. Sznitman R, Ali K, Richa R, Taylor RH, Hager GD, Fua P: Data-driven visual tracking in retinal microsurgery. Med Image Comput Comput Assist Intervent 2012:568–575, 2012. https://doi.org/10.1007/978-3-642-33418-4_70

    Article  Google Scholar 

  8. Pan SJ, Yang Q: A survey on transfer learning. IEEE Trans Knowl Data Eng 22:1345–1359, 2010. https://doi.org/10.1109/TKDE.2009.191

    Article  Google Scholar 

  9. Long J, Shelhamer E, Darrell T: Fully convolutional networks for semantic segmentation, in: 2015 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE, 2015: pp. 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

  10. Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition, (2014). http://arxiv.org/abs/1409.1556 (accessed September 28, 2018)

  11. He K, Zhang X, RenS, Sun J: Deep residual learning for image recognition, in: 2016 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE, 2016: pp. 770–778. https://doi.org/10.1109/CVPR.2016.90

  12. Attia M, HossnyM, Nahavandi S, Asadi H: Surgical tool segmentation using a hybrid deep CNN-RNN auto encoder-decoder, in: 2017 IEEE Int. Conf. Syst. Man, Cybern., IEEE, 2017: pp. 3373–3378. https://doi.org/10.1109/SMC.2017.8123151

  13. Garcia-Peraza-Herrera LC, Li W, Fidon L, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S: ToolNet: Holistically-nested real-time segmentation of robotic surgical tools, in: 2017 IEEE/RSJ Int. Conf. Intell. Robot. Syst., IEEE, 2017: pp. 5717–5722. https://doi.org/10.1109/IROS.2017.8206462

  14. García-Peraza-Herrera LC, Li W, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S: Real-time segmentation of non-rigid surgical tools based on deep learning and tracking, in: International Workshop on Computer-Assisted and Robotic Endoscopy (CARE), 2017: pp. 84–95. https://doi.org/10.1007/978-3-319-54057-3_8

    Chapter  Google Scholar 

  15. Ronneberger O, Fischer P, Brox T: U-Net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, 2015: pp. 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  16. Allan M, Shvets A, Kurmann T, Zhang Z, Duggal R, Su Y-H, Rieke N, Laina I, Kalavakonda N, Bodenstedt S, Herrera L, Li W, Iglovikov V, Luo H, Yang J, Stoyanov D, Maier-Hein L, Speidel S, Azizian M: 2017 Robotic Instrument Segmentation Challenge, 2019. http://arxiv.org/abs/1902.06426 (accessed 23 Feb 2019)

    Google Scholar 

  17. Yu F, Koltun V: Multi-Scale Context Aggregation by Dilated Convolutions, Eprint ArXiv:1511.07122. (2015). http://arxiv.org/abs/1511.07122 (accessed 28 Sept 2018)

  18. Xie S, Tu Z: Holistically-nested edge detection. Int J Comput Vis 125:3–18, 2017. https://doi.org/10.1007/s11263-017-1004-z

    Article  Google Scholar 

  19. Zeiler MD, Krishnan D, Taylor GW, Fergus R: Deconvolutional networks, in: 2010 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., IEEE, 2010: pp. 2528–2535. https://doi.org/10.1109/CVPR.2010.5539957

  20. Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X, Cottrell G: Understanding convolution for semantic segmentation, in: 2018 IEEE Winter Conf. Appl. Comput. Vis., IEEE, 2018: pp. 1451–1460. https://doi.org/10.1109/WACV.2018.00163

  21. Chang H-H, Zhuang AH, Valentino DJ, Chu W-C: Performance measure characterization for evaluating neuroimage segmentation algorithms. Neuroimage. 47:122–135, 2009. https://doi.org/10.1016/j.neuroimage.2009.03.068

    Article  PubMed  Google Scholar 

  22. Schaefer S, McPhail T, Warren J: Image deformation using moving least squares. ACM Trans Graph 25:533, 2006. https://doi.org/10.1145/1141911.1141920

    Article  Google Scholar 

  23. Kingma DP, Ba J: Adam: A method for stochastic Optimization, (2014). http://arxiv.org/abs/1412.6980 (accessed 4 Oct 2018)

  24. Shvets A, Rakhlin A, Kalinin A, Iglovikov V: Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning, (2018). https://arxiv.org/abs/1803.01207

    Google Scholar 

Download references

Author information

Affiliations

  1. College of Mechanical and Electrical Engineering, Harbin Engineering University, Harbin, Heilongjiang Province, China

    Lingtao Yu, Pengcheng Wang, Xiaoyan Yu, Yusheng Yan & Yongqiang Xia

Authors
  1. Lingtao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengcheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoyan Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yusheng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongqiang Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pengcheng Wang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yu, L., Wang, P., Yu, X. et al. A Holistically-Nested U-Net: Surgical Instrument Segmentation Based on Convolutional Neural Network. J Digit Imaging 33, 341–347 (2020). https://doi.org/10.1007/s10278-019-00277-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-019-00277-1

Keywords

  • Surgical instrument segmentation
  • Deep learning
  • Convolutional neural network