Evaluating Surgical Skills from Kinematic Data Using Convolutional Neural Networks

  • Hassan Ismail FawazEmail author
  • Germain Forestier
  • Jonathan Weber
  • Lhassane Idoumghar
  • Pierre-Alain Muller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11073)


The need for automatic surgical skills assessment is increasing, especially because manual feedback from senior surgeons observing junior surgeons is prone to subjectivity and time consuming. Thus, automating surgical skills evaluation is a very important step towards improving surgical practice. In this paper, we designed a Convolutional Neural Network (CNN) to evaluate surgeon skills by extracting patterns in the surgeon motions performed in robotic surgery. The proposed method is validated on the JIGSAWS dataset and achieved very competitive results with 100% accuracy on the suturing and needle passing tasks. While we leveraged from the CNNs efficiency, we also managed to mitigate its black-box effect using class activation map. This feature allows our method to automatically highlight which parts of the surgical task influenced the skill prediction and can be used to explain the classification and to provide personalized feedback to the trainee.


Kinematic data RMIS Deep learning CNN 


  1. 1.
    Ahmidi, N., et al.: A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans. Biomed. Eng. 64(9), 2025–2041 (2017)CrossRefGoogle Scholar
  2. 2.
    Bridgewater, B., et al.: Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. Br. Med. J. 327(7405), 13–17 (2003)CrossRefGoogle Scholar
  3. 3.
    Fard, M.J., Ameri, S., Darin Ellis, R., Chinnam, R.B., Pandya, A.K., Klein, M.D.: Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int. J. Med. Robot. Comput. Assist. Surg. 14, e1850 (2018)Google Scholar
  4. 4.
    Forestier, G., Petitjean, F., Senin, P., Despinoy, F., Jannin, P.: Discovering discriminative and interpretable patterns for surgical motion analysis. In: Artificial Intelligence in Medicine, pp. 136–145 (2017)Google Scholar
  5. 5.
    Gao, Y., et al.: The JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: Modeling and Monitoring of Computer Assisted Interventions, MICCAI Workshop (2014)Google Scholar
  6. 6.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. Int. Conf. Artif. Intell. Stat. 9, 249–256 (2010)Google Scholar
  7. 7.
    Hatala, R., Cook, D.A., Brydges, R., Hawkins, R.: Constructing a validity argument for the objective structured assessment of technical skills (OSATS): a systematic review of validity evidence. Adv. Health Sci. Educ. 20(5), 1149–1175 (2015)CrossRefGoogle Scholar
  8. 8.
    Islam, G., Kahol, K., Li, B., Smith, M., Patel, V.L.: Affordable, web-based surgical skill training and evaluation tool. J. Biomed. Inform. 59, 102–114 (2016)CrossRefGoogle Scholar
  9. 9.
    Kassahun, Y., et al.: Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int. J. Comput. Assist. Radiol. Surg. 11(4), 553–568 (2016)CrossRefGoogle Scholar
  10. 10.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)Google Scholar
  11. 11.
    Maier-Hein, L., et al.: Surgical data science for next-generation interventions. Nat. Biomed. Eng. 1(9), 691–696 (2017)CrossRefGoogle Scholar
  12. 12.
    Niitsu, H., et al.: Using the objective structured assessment of technical skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surg. Today 43(3), 271–275 (2013)CrossRefGoogle Scholar
  13. 13.
    Tao, L., et al.: Sparse hidden Markov models for surgical gesture classification and skill evaluation. In: Abolmaesumi, P., Joskowicz, L., Navab, N., Jannin, P. (eds.) IPCAI 2012. LNCS, vol. 7330, pp. 167–177. Springer, Heidelberg (2012). Scholar
  14. 14.
    Tedesco, M.M., Pak, J.J., Harris, E.J., Krummel, T.M., Dalman, R.L., Lee, J.T.: Simulation-based endovascular skills assessment: the future of credentialing? J. Vasc. Surg. 47(5), 1008–1014 (2008)CrossRefGoogle Scholar
  15. 15.
    Polavarapu, V.: H., Kulaylat, A., Sun, S., Hamed, O.: 100 years of surgical education: the past, present, and future. Bull. Am. Coll. Surg. 98(7), 22–27 (2013)Google Scholar
  16. 16.
    Vedula, S.S., et al.: Analysis of the structure of surgical activity for a suturing and knot-tying task. Public Libr. Sci. One 11(3), 1–14 (2016)Google Scholar
  17. 17.
    Wang, Z., Yan, W., Oates, T.: Time series classification from scratch with deep neural networks: a strong baseline. In: International Joint Conference on Neural Networks, pp. 1578–1585 (2017)Google Scholar
  18. 18.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)Google Scholar
  19. 19.
    Zia, A., Essa, I.: Automated Surgical Skill Assessment in RMIS Training. ArXiv e-prints (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Hassan Ismail Fawaz
    • 1
    Email author
  • Germain Forestier
    • 1
  • Jonathan Weber
    • 1
  • Lhassane Idoumghar
    • 1
  • Pierre-Alain Muller
    • 1
  1. 1.IRIMAS, Université de Haute-AlsaceMulhouseFrance

Personalised recommendations