An Efficient One-Stage Detector for Real-Time Surgical Tools Detection in Robot-Assisted Surgery

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12722)


Robot-assisted surgery (RAS) is a type of minimally invasive surgery which is completely different from the traditional surgery. RAS reduces surgeon’s fatigue and the number of doctors participating in surgery. At the same time, it causes less pain and has a faster recovery rate. Real-time surgical tools detection is important for computer-assisted surgery because the prerequisite for controlling surgical tools is to know the location of surgical tools. In order to achieve comparable performance, most Convolutional Neural Network (CNN) employed for detecting surgical tools generate a huge number of feature maps from expensive operation, which results in redundant computation and long inference time. In this paper, we propose an efficient and novel CNN architecture which generate ghost feature maps cheaply based on intrinsic feature maps. The proposed detector is more efficient and simpler than the state-of-the-art detectors. We believe the proposed method is the first to generate ghost feature maps for detecting surgical tools. Experimental results show that the proposed method achieves 91.6% mAP on the Cholec80-locations dataset and 100% mAP on the Endovis Challenge dataset with the detection speed of 38.5 fps, and realizes real-time and accurate surgical tools detection in the Laparoscopic surgery video.


  1. 1.
    Choi, B., Jo, K., Choi, S., Choi, J.: Surgical-tools detection based on convolutional neural network in laparoscopic robot-assisted surgery. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), vol. 2017, pp. 1756–1759 (2017)Google Scholar
  2. 2.
    Liu, Y., Zhao, Z., Chang, F., Hu, S.: An anchor-free convolutional neural network for real-time surgical tool detection in robot-assisted surgery. IEEE Access. PP(99), 1 (2020)Google Scholar
  3. 3.
    Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1), 86–97 (2017)CrossRefGoogle Scholar
  4. 4.
    Hajj, H.A., Lamard, M., Conze, P.H., Cochener, B., Quellec, G.: Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. Med. Image Anal. 47, 203–218 (2018)CrossRefGoogle Scholar
  5. 5.
    Sahu, M., Moerman, D., Mewes, P., Mountney, P., Rose, G.: Instrument state recognition and tracking for effective control of robotized laparoscopic systems. Int. J. Mech. Eng. Robot. Res. 5(1), 33–38 (2016)Google Scholar
  6. 6.
    Du, X., et al.: Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery. Int. J. Comput. Assist. Radiol. Surg. 11(6), 1109–1119 (2016)CrossRefGoogle Scholar
  7. 7.
    Zhao, Z., Voros, S., Chen, Z., Cheng, X.: Surgical tool tracking based on two CNNs: from coarse to fine. The J. Eng. 2019(14), 467–472 (2019)CrossRefGoogle Scholar
  8. 8.
    Nwoye, C.I., Mutter, D., Marescaux, J., Padoy, N.: Weakly supervised convolutional lstm approach for tool tracking in laparoscopic videos. Int. J. Comput. Assist. Radiol. Surg. 14(6), 1059–1067 (2019)CrossRefGoogle Scholar
  9. 9.
    Garc´lła-Peraza-Herrera, L.C., et al.: Real-time segmentation of non-rigid surgical tools based on deep learning and tracking. In: International Workshop on Computer-Assisted and Robotic Endoscopy (2016)Google Scholar
  10. 10.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Computer Vision and Pattern Recognition (2016)Google Scholar
  11. 11.
    Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: optimal speed and accuracy of object detection (2020)Google Scholar
  12. 12.
    Shi, P., Zhao, Z., Hu, S., et al.: Real-time surgical tool detection in minimally invasive surgery based on attention-guided convolutional neural network. IEEE Access PP(99), 1–1 (2020)Google Scholar
  13. 13.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional neural networks. In: European Conference on Computer Vision (2013)Google Scholar
  14. 14.
    Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C.: Ghostnet: more features from cheap operations. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)Google Scholar
  15. 15.
    Liu, W., et al.: SSD: single shot multibox detector (2016)Google Scholar
  16. 16.
    Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Yeh, I.H.: CSPNet: a new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020)Google Scholar
  17. 17.
    Misra, D.: Mish: a self regularized non-monotonic activation function (2019)Google Scholar
  18. 18.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation (2015)Google Scholar
  19. 19.
    He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2014)CrossRefGoogle Scholar
  20. 20.
    Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv e-prints (2018)Google Scholar
  21. 21.
    Zheng, Z., Wang, P., Liu, W., Li, J., Ren, D.: Distance-IoU loss: faster and better learning for bounding box regression. In: AAAI Conference on Artificial Intelligence (2020)Google Scholar
  22. 22.
    Du, X., et al.: Articulated multi-instrument 2-d pose estimation using fully convolutional networks. IEEE Trans. Med. Imaging 37(5), 1276–1287 (2018)CrossRefGoogle Scholar
  23. 23.
    Neubeck, A., Gool, L.J.V.: Efficient Non-Maximum Suppression (2006)Google Scholar
  24. 24.
    Rezatofighi, H., Tsoi, N., Gwak, J.Y., Sadeghian, A., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  25. 25.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2016)Google Scholar
  26. 26.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Comput. Sci. (2014) Google Scholar

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.School of Control Science and EngineeringShandong UniversityJinanChina
  2. 2.Department of General SurgeryFirst Affiliated Hospital of Shandong First Medical UniversityJinanChina

Personalised recommendations