Towards Skill Transfer via Learning-Based Guidance in Human-Robot Interaction: An Application to Orthopaedic Surgical Drilling Skill

  • Ehsan Zahedi
  • Fariba Khosravian
  • Weiqi Wang
  • Mehran Armand
  • Javad Dargahi
  • Mehrdad ZadehEmail author


This paper presents a machine learning-based guidance (LbG) approach for kinesthetic human-robot interaction (HRI) that can be used in virtual training simulations. Demonstrated positional and force skills are learned to both discriminate the skill levels of users and produce LbG forces. Force information is obtained from virtual forces, which developed based on real computed tomography (CT) data, rather than force sensors. A femur bone drilling simulation is developed to provide a practice environment for orthopaedic residents. The residents are provided with haptic feedback that enable them to feel the variable stiffness of bone layers. The X-ray views of the bone are also presented to them for better tracking of a pre-defined path inside the bone. The simulation is capable of planning a drill path, generating X-rays based on user defined orientation, and recording motion data for user assessment and skill modeling. The knowledge of expert surgeons is also incorporated into the simulation to provide LbG forces for improving the unpredictable motions of the residents. To discriminate the skill level of users, machine learning tools are used to develop surgical expert and resident models. In addition, to improve residents performance, the expert HCRF is used to generate adaptive LbG forces regarding the similarities between residents motions and the expert model. Experimental results show that the learning-based approach is able to assess the skill of users and improve residents performance.


Human-robot interaction Machine learning-based guidance Virtual surgical simulation 

JEL Classification

68T40 93C85 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



We thank Russell H. Taylor for facilitating this research in the LCSR lab at Johns Hopkins University.


  1. 1.
    Schapira, D., Schapira, C.: Osteoporosis: the evolution of a scientific term. Osteoporos. Int. 2(4), 164 (1992)CrossRefGoogle Scholar
  2. 2.
    Wade, S., Strader, C., Fitzpatrick, L., Anthony, M., O’Malley, C.: Estimating prevalence of osteoporosis: examples from industrialized countries. Arch. Osteoporos. 9(1), 182 (2014)CrossRefGoogle Scholar
  3. 3.
    Goldacre, M.J., Roberts, S.E., Yeates, D.: Mortality after admission to hospital with fractured neck of femur: database study. Bmj 325(7369), 868 (2002)CrossRefGoogle Scholar
  4. 4.
    Thorngren, K.G.: National Registration of Hip Fractures (2008)Google Scholar
  5. 5.
    Coles, T., Meglan, D., John, N.: The role of haptics in medical training simulators : a survey of the state of the art. IEEE Trans. Haptic 4(1), 51 (2011)CrossRefGoogle Scholar
  6. 6.
    Ahlberg, G.: The role of simulation technology for skills acquisition in image guided surgery. Institutionen för kirurgisk vetenskap/Department of Surgical Science (2005)Google Scholar
  7. 7.
    Seymour, N.: VR To OR: a review of the evidence that virtual reality simulation improves operating room performance. World J. Surg. 32(2), 182 (2008)CrossRefGoogle Scholar
  8. 8.
    Zahedi, E., Dargahi, J., Kia, M., Zadeh, M.: Gesture-based adaptive haptic guidance: a comparison of discriminative and generative modeling approaches. IEEE Robotics and Automation Letters 2(2), 1015 (2017)CrossRefGoogle Scholar
  9. 9.
    Rosen, J., Brown, J., Chang, L., Sinanan, M., Hannaford, B.: Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Trans. Biomed. Eng. 53(3), 399 (2006)CrossRefGoogle Scholar
  10. 10.
    Reiley, C., Hager, G.: Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 435–442 (2009)Google Scholar
  11. 11.
    Kahol, K., Vankipuram, M., Smith, M.: Cognitive simulators for medical education and training. J. Biomed. Inform. 42(4), 593 (2009)CrossRefGoogle Scholar
  12. 12.
    Chi, W., Rafii-Tari, H., Payne, C.J., Liu, J., Riga, C., Bicknell, C., Yang, G.Z.: A learning based training and skill assessment platform with haptic guidance for endovascular catheterization. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 2357–2363, IEEE (2017)Google Scholar
  13. 13.
    Lee, D., Ott, C.: Incremental kinesthetic teaching of motion primitives using the motion refinement tube. Auton. Robot. 31(2-3), 115 (2011)CrossRefGoogle Scholar
  14. 14.
    Medina, J., Lee, D., Hirche, S.: Risk-Sensitive Optimal feedback control for haptic assistance. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1025–1031 (2012)Google Scholar
  15. 15.
    Rozo, L., Jiménez, P., Torras, C.: A robot learning from demonstration framework to perform force-based manipulation tasks. Intell. Serv. Robot. 6(1), 33 (2013)CrossRefGoogle Scholar
  16. 16.
    Rozo, L., Calinon, S., Caldwell, D.G., Jimenez, P., Torras, C.: Learning physical collaborative robot behaviors from human demonstrations. IEEE Trans. Robot. 32(3), 513 (2016)CrossRefGoogle Scholar
  17. 17.
    Kronander, K., Billard, A.: Learning compliant manipulation through kinesthetic and tactile human-robot interaction. IEEE Trans. Haptic 7(3), 367 (2014)CrossRefGoogle Scholar
  18. 18.
    Bernardino, A., Henriques, M., Hendrich, N., Zhang, J.: Precision grasp synergies for dexterous robotic hands. In: IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 62–67 (2013)Google Scholar
  19. 19.
    Calinon, S., Billard, A.: Incremental learning of gestures by imitation in a humanoid robot. In: Proceedings of the ACM/IEEE international conference on Human-robot interaction, pp. 255–262 (2007)Google Scholar
  20. 20.
    Cha, E., Kronander, K., Billard, A.: Combined kinesthetic and simulated interface for teaching robot motion models. In: IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 83–88 (2015)Google Scholar
  21. 21.
    Aleotti, J., Caselli, S., Reggiani, M.: Leveraging on a virtual environment for robot programming by demonstration. Robot. Auton. Syst. 47(2), 153 (2004)CrossRefGoogle Scholar
  22. 22.
    Aleotti, J., Caselli, S., Reggiani, M.: Evaluation of virtual fixtures for a robot programming by demonstration interface. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 35(4), 536 (2005)CrossRefGoogle Scholar
  23. 23.
    Koropouli, V., Lee, D., Hirche, S.: Learning interaction control policies by demonstration. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 344–349 (2011)Google Scholar
  24. 24.
    Kormushev, P., Calinon, S., Caldwell, D.G.: Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input. Adv. Robot. 25(5), 581 (2011)CrossRefGoogle Scholar
  25. 25.
    Medina, J., Lorenz, T., Hirche, S.: Synthesizing anticipatory haptic assistance considering human behavior uncertainty. IEEE Trans. Robot. 31(1), 180 (2015)CrossRefGoogle Scholar
  26. 26.
    Wang, S., Quattoni, A., Morency, L., Demirdjian, D., Darrell, T.: Hidden conditional random fields for gesture recognition. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2, 1521 (2006)Google Scholar
  27. 27.
    Lu, W., Tong, Z., Chu, J.: Dynamic hand gesture recognition with leap motion controller. IEEE Signal Process Lett. 23(9), 1188 (2016)CrossRefGoogle Scholar
  28. 28.
    Arzani, M.M., Fathy, M., Aghajan, H., Azirani, A.A., Raahemifar, K., Adeli, E.: Structured prediction with short/long-range dependencies for human activity recognition from depth skeleton data. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 560–567 (2017)Google Scholar
  29. 29.
    Liu, A.A., Nie, W.Z., Su, Y.T., Ma, L., Hao, T., Yang, Z.X.: Coupled hidden conditional random fields for rgb-d human action recognition. Signal Process. 112, 74 (2015)CrossRefGoogle Scholar
  30. 30.
    Hong, W.T.: HCRF-Based model compensation for noisy speech recognition. In: IEEE International Symposium on Consumer Electronics (ISCE), pp. 277–278 (2013)Google Scholar
  31. 31.
    Barriere, V., Clavel, C., Essid, S.: Opinion dynamics modeling for movie review transcripts classification with hidden conditional random fields. arXiv:1806.07787 (2018)
  32. 32.
    Nocedal, J., Wright, S.: Numerical optimization. Springer Science & Business Media (2006)Google Scholar
  33. 33.
    Smid, M.: Handbook of computational geometry (2000)Google Scholar
  34. 34.
    Diolaiti, N., Niemeyer, G., Barbagli, F., Salisbury, J.K.: Stability of haptic rendering: Discretization, quantization, time delay, and coulomb effects. IEEE Trans. Robot. 22(2), 256 (2006)CrossRefGoogle Scholar
  35. 35.
    Adams, R.J., Hannaford, B.: Stable haptic interaction with virtual environments. IEEE Trans. Robot. Autom. 15(3), 465 (1999)CrossRefGoogle Scholar
  36. 36.
    Teo, J.C., Si-Hoe, K.M., Keh, J.E., Teoh, S.H.: Relationship between ct intensity, micro-architecture and mechanical properties of porcine vertebral cancellous bone. Clin. Biomech. 21(3), 235 (2006)CrossRefGoogle Scholar
  37. 37.
    Sofronia, R.E., Davidescu, A., Savii, G.G.: Towards a virtual reality simulator for orthognathic basic skils. In: Applied Mechanics and Materials, vol. 162, pp. 352–357 (2012)Google Scholar
  38. 38.
    Bogoni, T.N., Pinho, M.S: Haptic technique for simulating multiple density materials and material removal. In: International Conference on Computer Graphics Visualization and Computer Vision, pp. 151–160 (2013)Google Scholar
  39. 39.
    Liu, Y., Laycock, S.D.: A haptic system for drilling into volume data with Polygonal Tools. In: TPCG, pp. 9–16 (2009)Google Scholar
  40. 40.
    Morris, D., Sewell, C., Blevins, N., Barbagli, F., Salisbury, K.: A collaborative virtual environment for the simulation of temporal bone surgery. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 319–327 (2004)Google Scholar
  41. 41.
    Müller, M. E., Allgöwer, M., Perren, S.: Manual of Internal Fixation: Techniques Recommended by the AO-ASIF Group. Springer Science & Business Media, Berlin (1991)CrossRefGoogle Scholar
  42. 42.
    Brown, T.D., Ferguson, A.B.: Mechanical property distributions in the cancellous bone of the human proximal femur. Acta Orthop. Scand. 51(1-6), 429 (1980)CrossRefGoogle Scholar
  43. 43.
    Pandey, R.K., Panda, S.: Drilling of bone: a comprehensive review. Journal of Clinical Orthopaedics and Trauma 4(1), 15 (2013)CrossRefGoogle Scholar
  44. 44.
    Boner, V., Kuhn, P., Mendel, T., Gisep, A.: Temperature evaluation during pmma screw augmentation in osteoporotic bone—an in vitro study about the risk of thermal necrosis in human femoral heads. J. Biomed. Mater. Res. B Appl. Biomater. 90(2), 842 (2009)CrossRefGoogle Scholar
  45. 45.
    Pourkand, A., Salas, C., Regalado, J., Bhakta, K., Tufaro, R., Mercer, D., Grow, D.: Objective evaluation of motor skills for orthopedic residents using a motion tracking drill system: Outcomes of an abos approved surgical skills training program. Iowa Orthop. J. 36, 13 (2016)Google Scholar
  46. 46.
    Pettersson, J., Palmerius, K.L., Knutsson, H., Wahlstrom, O., Tillander, B., Borga, M.: Simulation of patient specific cervical hip fracture surgery with a volume haptic interface. IEEE Trans. Biomed. Eng. 55(4), 1255 (2008)CrossRefGoogle Scholar
  47. 47.
    Reiley, C.E., Plaku, E., Hager, G.D.: Motion generation of robotic surgical tasks : learning from expert demonstrations. Engineering in Medicine and Biology Society, pp. 967–970 (2010)Google Scholar
  48. 48.
    Chmarra, M.K., Klein, S., de Winter, J.C., Jansen, F.W., Dankelman, J.: Objective classification of residents based on their psychomotor laparoscopic skills. Surg. Endosc. 24(5), 1031 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Mechanical, Industrial, Aerospace EngineeringConcordia UniversityMontrealCanada
  2. 2.Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations