Annals of Biomedical Engineering

, Volume 46, Issue 10, pp 1450–1464 | Cite as

Toward Improving Safety in Neurosurgery with an Active Handheld Instrument

  • Sara Moccia
  • Simone Foti
  • Arpita Routray
  • Francesca Prudente
  • Alessandro Perin
  • Raymond F. Sekula
  • Leonardo S. Mattos
  • Jeffrey R. Balzer
  • Wendy Fellows-Mayle
  • Elena De Momi
  • Cameron N. RiviereEmail author
Medical Robotics


Microsurgical procedures, such as petroclival meningioma resection, require careful surgical actions in order to remove tumor tissue, while avoiding brain and vessel damaging. Such procedures are currently performed under microscope magnification. Robotic tools are emerging in order to filter surgeons’ unintended movements and prevent tools from entering forbidden regions such as vascular structures. The present work investigates the use of a handheld robotic tool (Micron) to automate vessel avoidance in microsurgery. In particular, we focused on vessel segmentation, implementing a deep-learning-based segmentation strategy in microscopy images, and its integration with a feature-based passive 3D reconstruction algorithm to obtain accurate and robust vessel position. We then implemented a virtual-fixture-based strategy to control the handheld robotic tool and perform vessel avoidance. Clay vascular phantoms, lying on a background obtained from microscopy images recorded during petroclival meningioma surgery, were used for testing the segmentation and control algorithms. When testing the segmentation algorithm on 100 different phantom images, a median Dice similarity coefficient equal to 0.96 was achieved. A set of 25 Micron trials of 80 s in duration, each involving the interaction of Micron with a different vascular phantom, were recorded, with a safety distance equal to 2 mm, which was comparable to the median vessel diameter. Micron’s tip entered the forbidden region 24% of the time when the control algorithm was active. However, the median penetration depth was 16.9 μm, which was two orders of magnitude lower than median vessel diameter. Results suggest the system can assist surgeons in performing safe vessel avoidance during neurosurgical procedures.


Robot-assisted surgery Vessel segmentation Virtual fixture control Neurosurgery 



Partial funding provided by U.S. National Institutes of Health (Grant No. R01EB000526).


  1. 1.
    Abadi, M., A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint, 2016. arXiv: 1603.04467.
  2. 2.
    Al-Mefty, O., J. L. Fox, Sr., and R. R. Smith. Petrosal approach for petroclival meningiomas. Neurosurgery 22(3):510–517, 1988.CrossRefGoogle Scholar
  3. 3.
    Bay, H., T. Tuytelaars, and L. Van Gool. SURF: Speeded up robust features. European Conference on Computer Vision, 2006, pp. 404–417.Google Scholar
  4. 4.
    Becker, B. C., R. A. MacLachlan, L. A. Lobes, G. D. Hager, and C. N. Riviere. Vision-based control of a handheld surgical micromanipulator with virtual fixtures. IEEE Trans. Robot. 29(3):674–683, 2013.CrossRefGoogle Scholar
  5. 5.
    Beretta, E., E. De Momi, F. Rodriguez y Baena, and G. Ferrigno. Adaptive hands-on control for reaching and targeting tasks in surgery. Int. J. Adv. Robot. Syst. 12(5):50, 2015.CrossRefGoogle Scholar
  6. 6.
    Beretta, E., G. Ferrigno, and E. De Momi. Nonlinear force feedback enhancement for cooperative robotic neurosurgery enforces virtual boundaries on cortex surface. J. Med. Robot. Res. 1(02):1650001, 2016.Google Scholar
  7. 7.
    Braun, D., S. Yang, J. N. Martel, C. N. Riviere, and B. C. Becker. EyeSLAM: real-time simultaneous localization and mapping of retinal vessels during intraocular microsurgery. Int. J. Med. Robot. Comput. Assist. Surg. 14(1):e1848, 2017.CrossRefGoogle Scholar
  8. 8.
    Chan, T. F., and L. A. Vese. Active contours without edges. Trans. Image Process. 10(2):266–277, 2001.CrossRefGoogle Scholar
  9. 9.
    Chaudhuri, S., S. Chatterjee, N. Katz, M. Nelson, and M. Goldbaum. Detection of blood vessels in retinal images using two-dimensional matched filters. IEEE Trans. Med. Imaging 8(3):263–269, 1989.CrossRefGoogle Scholar
  10. 10.
    Cheng, Y., X. Hu, J. Wang, Y. Wang and S. Tamura. Accurate vessel segmentation with constrained B-snake. IEEE Trans. Med. Imaging 24(8):2440–2455, 2015.CrossRefGoogle Scholar
  11. 11.
    Dasgupta, A., and S. Singh. A fully convolutional neural network based structured prediction approach towards the retinal vessel segmentation. In: IEEE International Symposium on Biomedical Imaging, 2017, pp. 248–251.Google Scholar
  12. 12.
    DiLuna, M. L., and K. R. Bulsara. Surgery for petroclival meningiomas: a comprehensive review of outcomes in the skull base surgery era. Skull Base 20(05):337–342, 2010.CrossRefGoogle Scholar
  13. 13.
    Durrant-Whyte, H., and T. Bailey. Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag. 13(2):99–110, 2006.CrossRefGoogle Scholar
  14. 14.
    El-Manadili, Y., and K. Novak. Precision rectification of SPOT imagery using the direct linear transformation model. Photogram. Eng. Remot. Sens. 62(1):67–72, 1996.Google Scholar
  15. 15.
    Esteva, A., B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115–118, 2017.CrossRefGoogle Scholar
  16. 16.
    Felzenszwalb, P., and D. Huttenlocher. Distance transforms of sampled functions. Tech. Rep., Cornell University, 2004.Google Scholar
  17. 17.
    Frangi, A. F., W. J. Niessen, K. L. Vincken, and M. A. Viergever. Multiscale vessel enhancement filtering. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 1998, pp. 130–137.Google Scholar
  18. 18.
    Fraz, M. M., P. Remagnino, A. Hoppe, B. Uyyanonvara, A. R. Rudnicka, C. G. Owen, and S. A. Barman. Blood vessel segmentation methodologies in retinal images—a survey. Comput. Methods Progr. Biomed. 108(1):407–433, 2012.CrossRefGoogle Scholar
  19. 19.
    Fu, H., Wang, C., Tao, D., Black, M.J.: Occlusion boundary detection via deep exploration of context. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 241–250 (2016)Google Scholar
  20. 20.
    Fu, H., Y. Xu, S. Lin, D. W. K. Wong, and J. Liu. DeepVessel: retinal vessel segmentation via deep learning and conditional random field. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2016, pp. 132–139.Google Scholar
  21. 21.
    Ganin, Y., and V. Lempitsky. N4-fields: neural network nearest neighbor fields for image transforms. Asian Conference on Computer Vision. Berlin: Springer, 2014, pp. 536–551.Google Scholar
  22. 22.
    Gijbels, A., E. B. Vander Poorten, P. Stalmans, and D. Reynaerts. Development and experimental validation of a force sensing needle for robotically assisted retinal vein cannulations. IEEE International Conference on Robotics and Automation, pp. 2270–2276, 2015.Google Scholar
  23. 23.
    Girshick, R., J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. Conference on Computer Vision and Pattern Recognition, pp. 580–587, 2014.Google Scholar
  24. 24.
    Gonenc, B., M. A. Balicki, J. Handa, P. Gehlbach, C. N. Riviere, R. H. Taylor, and I. Iordachita. Preliminary evaluation of a micro-force sensing handheld robot for vitreoretinal surgery. International Conference on Intelligent Robots and Systems, pp. 4125–4130, 2012.Google Scholar
  25. 25.
    Hackethal, A., M. Koppan, K. Eskef, and H. R. Tinneberg. Handheld articulating laparoscopic instruments driven by robotic technology. First clinical experience in gynecological surgery. Gynecol. Surg. 9(2):203, 2011.CrossRefGoogle Scholar
  26. 26.
    Kingma, D., and J. Ba. Adam: a method for stochastic optimization. arXiv preprint, 2014. arXiv: 1412.6980.
  27. 27.
    Krizhevsky, A., I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25:1097–1105, 2012.Google Scholar
  28. 28.
    Kwok, K. W., K. H. Tsoi, V. Vitiello, J. Clark, G. C. Chow, W. Luk, and G. Z. Yang. Dimensionality reduction in controlling articulated snake robot for endoscopy under dynamic active constraints. IEEE Trans. Robot. 29(1):15–31, 2013.CrossRefGoogle Scholar
  29. 29.
    Lam, L., S. W. Lee, and C. Y. Suen. Thinning methodologies—a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 14(9):869–885, 1992.CrossRefGoogle Scholar
  30. 30.
    Lang, J. Clinical Anatomy of the Head: Neurocranium, Orbit, Craniocervical Regions. Berlin: Springer, 2012.Google Scholar
  31. 31.
    Lawrence, J. D., A. M. Frederickson, Y. F. Chang, P. M. Weiss, P. C. Gerszten, and R. F. Sekula, Jr. An investigation into quality of life improvement in patients undergoing microvascular decompression for hemifacial spasm. J. Neurosurg. 128(1):193–201, 2017.Google Scholar
  32. 32.
    LeCun, Y., Y. Bengio, and G. Hinton. Deep learning. Nature 521(7553):436, 2015.CrossRefGoogle Scholar
  33. 33.
    Lee, K. H., Z. Guo, G. C. Chow, Y. Chen, W. Luk, and K. W. Kwok. GPU-based proximity query processing on unstructured triangular mesh model. IEEE International Conference on Robotics and Automation, 2015, pp. 4405–4411.Google Scholar
  34. 34.
    Li, Q., B. Feng, L. Xie, P. Liang, H. Zhang, and T. Wang. A cross-modality learning approach for vessel segmentation in retinal images. IEEE Trans. Med. Imaging 35(1):109–118, 2016.CrossRefGoogle Scholar
  35. 35.
    Liskowski, P., and K. Krawiec. Segmenting retinal blood vessels with deep neural networks. IEEE Trans. Med. Imaging 35(11):2369–2380, 2016.CrossRefGoogle Scholar
  36. 36.
    Liu, N., H. Li, M. Zhang, J. Liu, Z. Sun, and T. Tan. Accurate iris segmentation in non-cooperative environments using fully convolutional networks. IEEE International Conference on Biometrics, 2016, pp. 1–8.Google Scholar
  37. 37.
    Lonner, J. H. Robotically assisted unicompartmental knee arthroplasty with a handheld image-free sculpting tool. Orthop. Clin. N. Am. 47(1):29–40, 2016.CrossRefGoogle Scholar
  38. 38.
    MacLachlan, R. A., B. C. Becker, J. C. Tabarés, G. W. Podnar, L. A. Lobes, Jr, and C. N. Riviere. Micron: an actively stabilized handheld tool for microsurgery. IEEE Trans. Robot. 28(1):195–212, 2012.CrossRefGoogle Scholar
  39. 39.
    MacLachlan, R. A., and C. N. Riviere. High-speed microscale optical tracking using digital frequency-domain multiplexing. IEEE Trans. Instrum. Meas. 58(6):1991–2001, 2009.CrossRefGoogle Scholar
  40. 40.
    Maninis, K. K., J. Pont-Tuset, P. Arbeláez, and L. Van Gool. Deep retinal image understanding. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2016, pp. 140–148.Google Scholar
  41. 41.
    Merkow, J., A. Marsden, D. Kriegman, and Z. Tu. Dense volume-to-volume vascular boundary detection. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2016, pp. 371–379.Google Scholar
  42. 42.
    Michalak, S. M., J. D. Rolston, and M. T. Lawton. Incidence and predictors of complications and mortality in cerebrovascular surgery: National trends from 2007 to 2012. Neurosurgery 79(2):182–193, 2016.CrossRefGoogle Scholar
  43. 43.
    Mo, J., and L. Zhang. Multi-level deep supervised networks for retinal vessel segmentation. Int. J. Comput. Assist. Radiol. Surg. 12(12):2181–2193, 2017.CrossRefGoogle Scholar
  44. 44.
    Moccia, S., E. De Momi, S. El Hadji, and L. S. Mattos. Blood vessel segmentation algorithms – Review of methods, datasets and evaluation metrics. Comput. Methods Progr. Biomed. 158:71–91, 2018.CrossRefGoogle Scholar
  45. 45.
    Moccia, S., G. O. Vanone, E. De Momi, A. Laborai, L. Guastini, G. Peretti, and L. S. Mattos. Learning-based classification of informative laryngoscopic frames. Comput. Methods Progr. Biomed. 158:21–30, 2018.CrossRefGoogle Scholar
  46. 46.
    Morita, A., S. Sora, M. Mitsuishi, S. Warisawa, K. Suruman, D. Asai, J. Arata, S. Baba, H. Takahashi, R. Mochizuki, et al. Microsurgical robotic system for the deep surgical field: development of a prototype and feasibility studies in animal and cadaveric models. J. Neurosurg. 103(2):320–327, 2005.CrossRefGoogle Scholar
  47. 47.
    Motkoski, J. W., and G. R. Sutherland. Why robots entered neurosurgery. Exp. Neurosurg. Anim. Models 116:85–105, 2016.Google Scholar
  48. 48.
    Muja, M., and D. G. Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications 2(331-340), 2, 2009.Google Scholar
  49. 49.
    Nah, S., T. H. Kim, and K. M. Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. IEEE Conference on Computer Vision and Pattern Recognition, 2017.Google Scholar
  50. 50.
    Niu, P. P., Y. Yu, H. W. Zhou, Y. Liu, Y. Luo, Z. N. Guo, H. Jin, and Y. Yang. Vessel wall differences between middle cerebral artery and basilar artery plaques on magnetic resonance imaging. Sci. Rep. 6:38534 (2016)Google Scholar
  51. 51.
    Pan, J., L. Zhang, and D. Manocha. Collision-free and smooth trajectory computation in cluttered environments. Int. J. Robot. Res. 31(10):1155–1175, 2012.CrossRefGoogle Scholar
  52. 52.
    Poplin, R., A. V. Varadarajan, K. Blumer, Y. Liu, M. V. McConnell, G. S. Corrado, L. Peng, and D. R. Webster. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2:158–164, 2018.CrossRefGoogle Scholar
  53. 53.
    Prentašić, P., M. Heisler, Z. Mammo, S. Lee, A. Merkur, E. Navajas, M. F. Beg, M. Šarunić, and S. Lončarić. Segmentation of the foveal microvasculature using deep learning networks. J. Biomed. Opt. 21(7):075008, 2016.CrossRefGoogle Scholar
  54. 54.
    Prudente, F., S. Moccia, A. Perin, R. Sekula, L. Mattos, J. Balzer, W. Fellows-Mayle, E. De Momi, and C. Riviere. Toward safety enhancement in neurosurgery using a handheld robotic instrument. The Hamlyn Symposium on Medical Robotics, 2017, pp. 15–16.Google Scholar
  55. 55.
    Ronneberger, O., P. Fischer, and T. Brox. U-net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2015, pp. 234–241.Google Scholar
  56. 56.
    Salem, N., S. Salem, and A. Nandi. Segmentation of retinal blood vessels based on analysis of the Hessian matrix and clustering algorithm. European Signal Processing Conference, 2007, pp. 428–432.Google Scholar
  57. 57.
    Sekula, R. F., A. M. Frederickson, G. D. Arnone, M. R. Quigley, and M. Hallett. Microvascular decompression for hemifacial spasm in patients> 65 years of age: an analysis of outcomes and complications. Muscle Nerve 48(5):770–776, 2013.CrossRefGoogle Scholar
  58. 58.
    Smistad, E., and L. Løvstakken. Vessel detection in ultrasound images using deep convolutional neural networks. International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, pp. 30–38. Springer (2016)Google Scholar
  59. 59.
    Song, J., B. Gonenc, J. Guo, and I. Iordachita. Intraocular snake integrated with the steady-hand eye robot for assisted retinal microsurgery. IEEE International Conference on Robotics and Automation, 2017, pp. 6724–6729.Google Scholar
  60. 60.
    Sutherland, G. R., P. B. McBeth, and D. F. Louw. Neuroarm: an MR compatible robot for microsurgery. International Congress Series, vol. 1256. Amsterdam: Elsevier, 2003, pp. 504–508.Google Scholar
  61. 61.
    Sutherland, G. R., S. Wolfsberger, S. Lama, and K. Zareinia. The evolution of neuroArm. Neurosurgery 72(Suppl 1):A27–A32, 2013.CrossRefGoogle Scholar
  62. 62.
    Taylor, R. H., A. Menciassi, G. Fichtinger, P. Fiorini, and P. Dario. Medical robotics and computer-integrated surgery. In: Springer Handbook of Robotics. Cham: Springer, 2016, pp. 1657–1684.Google Scholar
  63. 63.
    Twinanda, A. P., S. Shehata, D. Mutter, J. Marescaux, M. de Mathelin, and N. Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1):86–97, 2017.CrossRefGoogle Scholar
  64. 64.
    Wang, S., Y. Yin, G. Cao, B. Wei, Y. Zheng, and G. Yang. Hierarchical retinal blood vessel segmentation based on feature and ensemble learning. Neurocomputing 149:708–717, 2015.CrossRefGoogle Scholar
  65. 65.
    Xue, D. X., R. Zhang, H. Feng, and Y. L. Wang. Cnn-SVM for microvascular morphological type recognition with data augmentation. J. Med. Biol. Eng. 36(6):755–764, 2016.CrossRefGoogle Scholar
  66. 66.
    Yang, S., R. A. MacLachlan, and C. N. Riviere. Design and analysis of 6 DOF handheld micromanipulator. IEEE International Conference on Robotics and Automation, 2012, pp. 1946–1951.Google Scholar
  67. 67.
    Yang, S., R. A. MacLachlan, and C. N. Riviere. Toward automated intraocular laser surgery using a handheld micromanipulator. IEEE International Conference on Intelligent Robots and Systems, 2014, pp. 1302–1307.Google Scholar
  68. 68.
    Yang, S., R. A. MacLachlan, and C. N. Riviere. Manipulator design and operation of a six-degree-of-freedom handheld tremor-canceling microsurgical instrument. IEEE/ASME Trans. Mechatron. 20(2):761–772, 2015.CrossRefGoogle Scholar
  69. 69.
    Zhang, B., L. Zhang, L. Zhang, and F. Karray. Retinal vessel extraction by matched filter with first-order derivative of Gaussian. Comput. Biol. Med. 40(4):438–445, 2010.CrossRefGoogle Scholar

Copyright information

© Biomedical Engineering Society 2018

Authors and Affiliations

  • Sara Moccia
    • 1
    • 2
  • Simone Foti
    • 2
  • Arpita Routray
    • 5
  • Francesca Prudente
    • 2
  • Alessandro Perin
    • 3
  • Raymond F. Sekula
    • 4
  • Leonardo S. Mattos
    • 1
  • Jeffrey R. Balzer
    • 4
  • Wendy Fellows-Mayle
    • 4
  • Elena De Momi
    • 2
  • Cameron N. Riviere
    • 5
    Email author
  1. 1.Department of Advanced RoboticsIstituto Italiano di TecnologiaGenoaItaly
  2. 2.Department of Electronics, Information and BioengineeringPolitecnico di MilanoMilanItaly
  3. 3.Besta NeuroSim CenterIRCCS Istituto Neurologico C. BestaMilanItaly
  4. 4.Department of Neurological SurgeryUniversity of PittsburghPittsburghUSA
  5. 5.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations