Autonomous Robots

, Volume 42, Issue 5, pp 997–1009 | Cite as

Skill-based human–robot cooperation in tele-operated path tracking

  • Nima Enayati
  • Giancarlo Ferrigno
  • Elena De Momi
Part of the following topical collections:
  1. Special Issue: Learning for Human-Robot Collaboration


This work proposes a shared-control tele-operation framework that adapts its cooperative properties to the estimated skill level of the operator. It is hypothesized that different aspects of an operator’s performance in executing a tele-operated path tracking task can be assessed through conventional machine learning methods using motion-based and task-related features. To identify performance measures that capture motor skills linked to the studied task, an experiment is conducted where users new to tele-operation, practice towards motor skill proficiency in 7 training sessions. A set of classifiers are then learned from the acquired data and selected features, which can generate a skill profile that comprises estimations of user’s various competences. Skill profiles are exploited to modify the behavior of the assistive robotic system accordingly with the objective of enhancing user experience by preventing unnecessary restriction for skilled users. A second experiment is implemented in which novice and expert users execute the path tracking on different pathways while being assisted by the robot according to their estimated skill profiles. Results validate the skill estimation method and hint at feasibility of shared-control customization in tele-operated path tracking.


Shared-control Active constraints Virtual fixtures Tele-operation Machine learning Surgery 



This work has received funding from the European Union’s Horizon 2020 research and innovation program under Grant agreement no. H2020-ICT-26-2016-1-732515 (SMARTsurg project).


  1. Abbott, J. J., Marayong, P., & Okamura A. M. (2007). Haptic virtual fixtures for robot-assisted manipulation. Robotics Research.
  2. Ahmidi, N., et al. (2017). A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Transactions on Biomedical Engineering, 9294(c), 1–16.Google Scholar
  3. Anderson, S. J., Karumanchi, S. B., & Iagnemma, K. (2012). Constraint-based planning and control for safe, semi-autonomous operation of vehicles. In 2012 IEEE intelligent vehicles symposium (pp. 383–88). IEEE.Google Scholar
  4. Bauzano, E., Estebanez, B., & Muñoz, V. F. (2014). Collaborative human-robot system for HALS suture procedures. IEEE Systems Journal, 10, 1–10.Google Scholar
  5. Beretta, E., et al. (2015). Enhanced torque-based impedance control to assist brain targeting during open-skull neurosurgery: A feasibility study. International Journal of Medical Robotics and Computer Assisted Surgery, 12(3), 326–41. Scholar
  6. Berthet-Rayne, P., Power, M., King, H., & Yang Fellow, G.-Z. (2016). Hubot?: A three state human–robot collaborative framework for bimanual surgical tasks based on learned models. In International conference on robotics and automation (pp. 715–722).Google Scholar
  7. Bowyer, S. A., Davies, B. L., & Baena, F. R. Y. (2014). Active constraints/virtual fixtures: A survey. IEEE Transactions on Robotics, 30(1), 138–57.CrossRefGoogle Scholar
  8. Bowyer, S. A., & Baena, F. R. Y. (2015). Dissipative control for physical human–robot interaction. IEEE Transactions on Robotics, 31(6), 1281–1293.CrossRefGoogle Scholar
  9. Bruyninckx, H. (2001). Open robot control software: The OROCOS project. In Proceedings 2001 ICRA. IEEE international conference on robotics and automation (Cat. No. 01CH37164) (pp. 2523–2528). IEEE.Google Scholar
  10. Chmarra, M. K., et al. (2010). Objective classification of residents based on their psychomotor laparoscopic skills. Surgical Endoscopy, 24(5), 1031–1039.CrossRefGoogle Scholar
  11. Cotin, S., et al. (2002). Metrics for laparoscopic skills trainers: The weakest link!. Medical Image Computing and Computer-Assisted Intervention—MICCAI, 2488, 35–43.zbMATHGoogle Scholar
  12. Dragan, A. D., & Srinivasa, S. S. (2013). A policy-blending formalism for shared control. The International Journal of Robotics Research, 32(7), 790–805.CrossRefGoogle Scholar
  13. Enayati, N., Costa, E. C. A., Ferrigno, G., & De Momi, E. (2016a). A dynamic non-energy-storing guidance constraint with motion redirection for robot-assisted surgery. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon, Korea: IEEE.Google Scholar
  14. Enayati, N., De Momi, E., & Ferrigno, G. (2016b). Haptics in robot-assisted surgery: Challenges and benefits. IEEE Reviews in Biomedical Engineering, 9, 49–65.Google Scholar
  15. Enayati, N., Ferrigno, G., & De Momi, E. (2017). Performance metrics for guidance active constraints in surgical robotics. The International Journal of Medical Robotics and Computer Assisted Surgery.
  16. Enayati, N., & Najafi, F. (2011). Design and manufacturing of a tele-operative rescue robot with a novel track arrangement. Industrial Robot: An International Journal, 38(5), 476–85.CrossRefGoogle Scholar
  17. Fard, M. J., Ameri, S., Chinnam, R. B., & Ellis, R. D. (2017). Soft boundary approach for unsupervised gesture segmentation in robotic-assisted surgery. IEEE Robotics and Automation Letters, 2(1), 171–178.CrossRefGoogle Scholar
  18. Finan, M. A., Clark, M. E., & Rocconi, R. P. (2010). A novel method for training residents in robotic hysterectomy. Journal of Robotic Surgery, 4(1), 33–39.CrossRefGoogle Scholar
  19. Franchi, A., et al. (2012). Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robotics & Automation Magazine, 19(3), 57–68.CrossRefGoogle Scholar
  20. Gao, M., Oberlander, J., Schamm, T., & Zollner, J. M. (2014). Contextual task-aware shared autonomy for assistive mobile robot teleoperation. In 2014 IEEE/RSJ international conference on intelligent robots and systems (pp. 3311–3318). IEEE.Google Scholar
  21. Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F. J., & Marín-Jiménez, M. J. (2014). Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6), 2280–2292.CrossRefGoogle Scholar
  22. Ghez, C., Krakauer, J. W., & Ghilardi, M.-F. (1999). Independent learning of internal models for kinematic and dynamic control of reaching. Nature Neuroscience, 2(11), 1026–1031.CrossRefGoogle Scholar
  23. Gomes, P. (2011). Surgical robotics: Reviewing the past, analysing the present, imagining the future. Robotics and Computer-Integrated Manufacturing, 27(2), 261–266.CrossRefGoogle Scholar
  24. Hajshirmohammadi, I., & Payandeh, S. (2007). Fuzzy set theory for performance evaluation in a surgical simulator. Presence: Teleoperators and Virtual Environments, 16(6), 603–622.CrossRefGoogle Scholar
  25. Van Hove, P. D., et al. (2010). Objective assessment of technical surgical skills. British Journal of Surgery, 97(7), 972–987.CrossRefGoogle Scholar
  26. Kim, S., et al. (2010). Lack of a discriminatory function for endoscopy skills on a computer-based simulator. Surgical Endoscopy, 24(12), 3008–3015. Scholar
  27. Kranzfelder, M., et al. (2013). Toward increased autonomy in the surgical OR: needs, requests, and expectations. Surgical Endoscopy and Other Interventional Techniques, 27(5), 1681–1688.CrossRefGoogle Scholar
  28. Lawrence, D. A. (1993). Stability and transparency in bilateral teleoperation. IEEE Transactions on Robotics and Automation, 9(5), 624–637.CrossRefGoogle Scholar
  29. Lin, H. C., Shafran, I., Yuh, D., & Hager, G. D. (2006). Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions. Computer Aided Surgery, 11(5), 220–230.CrossRefGoogle Scholar
  30. Liu, Y.-T., Mayer-Kress, G., & Newell, K. M. (2006). Qualitative and quantitative change in the dynamics of motor learning. Journal of Experimental Psychology: Human Perception and Performance, 32(2), 380–393.Google Scholar
  31. Marchal-Crespo, L., & Reinkensmeyer, D. J. (2008). Haptic guidance can enhance motor learning of a steering task. Journal of Motor Behavior, 40(6), 545–556.CrossRefGoogle Scholar
  32. Megali, G., Sinigaglia, S., Tonet, O., & Dario, P. (2006). Modelling and evaluation of surgical performance using hidden markov models. IEEE Transactions on Biomedical Engineering, 53(10), 1911–1919.CrossRefGoogle Scholar
  33. Milliken, L., & Hollinger, G. A. (2016). Modeling user expertise for choosing levels of shared autonomy. In Planning for human–robot interaction shared autonomy and collaborative robotics workshop, robotics: Science and systems conference.Google Scholar
  34. Nichols, K. A. et al. (2015). Models of human-centered automation in a debridement task. In IEEE international conference on intelligent robots and systems 2015–December (pp. 5784–5789).Google Scholar
  35. Padoy, N., Hager, G.D. (2011) Human-machine collaborative surgery using learned models. In Proceedings of the IEEE international conference on robotics and automation (pp. 5285–5292).Google Scholar
  36. Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825–2830.MathSciNetzbMATHGoogle Scholar
  37. Penin, L. F., Matsumoto, K., & Wakabayashi, S. (2000). Force reflection for time-delayed teleoperation of space robots. In Proceedings 2000 ICRA. Millennium conference. IEEE International conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065) (pp. 3120–3125). IEEE.Google Scholar
  38. Rafii-Tari, H., et al. (2013). Learning-based modeling of endovascular navigation for collaborative robotic catheterization. In International conference on medical image computing and computer-assisted intervention (pp. 369–377). Berlin, Heidelberg: Springer.Google Scholar
  39. Reiley, C. E., & Hager, G. D. (2009). Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. In International conference on medical image computing and computer-assisted intervention (pp. 435–442). Berlin, Heidelberg: Springer.Google Scholar
  40. Richards, C., et al. (2000). Skills evaluation in minimally invasive surgery using force/torque signatures. Surgical Endoscopy, 14(9), 791–798.CrossRefGoogle Scholar
  41. Rosen, J., et al. (2006). Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Transactions on Biomedical Engineering, 53(3), 399–413.CrossRefGoogle Scholar
  42. Sainburg, R. L., & Wang, J. (2002). Interlimb transfer of visuomotor rotations: Independence of direction and final position information. Experimental Brain Research, 145(4), 437–447.CrossRefGoogle Scholar
  43. Schmidt, R. A. (1991). Frequent augmented feedback can degrade learning: Evidence and interpretations. In: Tutorials in motor neuroscience (pp. 59–75). Dordrecht, Netherlands: Springer.Google Scholar
  44. Schreuder, H. W. R., et al. (2012). Training and learning robotic surgery, time for a more structured approach: A systematic review. BJOG: An International Journal of Obstetrics and Gynaecology, 119(2), 137–149.CrossRefGoogle Scholar
  45. Shadmehr, R., Smith, M. A., & Krakauer, J. W. (2010). Error correction, sensory prediction, and adaptation in motor control. Annual Review of Neuroscience, 33(1), 89–108.CrossRefGoogle Scholar
  46. Shmuelof, L., Krakauer, J. W., & Mazzoni, P. (2012). How is a motor skill learned? Change and invariance at the levels of task success and trajectory control. Journal of Neurophysiology, 108(2), 578–594.CrossRefGoogle Scholar
  47. Sigrist, R., Rauter, G., Riener, R., & Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic Bulletin & Review, 20(1), 21–53.CrossRefGoogle Scholar
  48. Takayama, L., Marder-Eppstein, E., Harris, H., & Beer, J. M. (2011). Assisted driving of a mobile remote presence system: System design and controlled user evaluation. In 2011 IEEE international conference on robotics and automation (pp. 1883–1889). IEEE.Google Scholar
  49. Trejos, A. L., et al. (2013). The application of force sensing to skills assessment in minimally invasive surgery. In 2013 IEEE international conference on robotics and automation (pp. 4370–4375). IEEE.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Department of Electronics, Information and BioengineeringPolitecnico di MilanoMilanItaly

Personalised recommendations