Automated video-based assessment of surgical skills for training and evaluation in medical schools

  • Aneeq ZiaEmail author
  • Yachna Sharma
  • Vinay Bettadapura
  • Eric L. Sarin
  • Thomas Ploetz
  • Mark A. Clements
  • Irfan Essa
Original Article



Routine evaluation of basic surgical skills in medical schools requires considerable time and effort from supervising faculty. For each surgical trainee, a supervisor has to observe the trainees in person. Alternatively, supervisors may use training videos, which reduces some of the logistical overhead. All these approaches however are still incredibly time consuming and involve human bias. In this paper, we present an automated system for surgical skills assessment by analyzing video data of surgical activities.


We compare different techniques for video-based surgical skill evaluation. We use techniques that capture the motion information at a coarser granularity using symbols or words, extract motion dynamics using textural patterns in a frame kernel matrix, and analyze fine-grained motion information using frequency analysis.


We were successfully able to classify surgeons into different skill levels with high accuracy. Our results indicate that fine-grained analysis of motion dynamics via frequency analysis is most effective in capturing the skill relevant information in surgical videos.


Our evaluations show that frequency features perform better than motion texture features, which in-turn perform better than symbol-/word-based features. Put succinctly, skill classification accuracy is positively correlated with motion granularity as demonstrated by our results on two challenging video datasets.


Surgical skill Classification Feature modeling 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Dennis BM, Long EL, Zamperini KM, Nakayama DK (2013) The effect of the 16-hour intern workday restriction on surgical residents’ in-hospital activities. J Surg Educ 70(6):800–805CrossRefPubMedGoogle Scholar
  2. 2.
    Awad S, Liscum K, Aoki N, Awad S, Berger D (2002) Does the subjective evaluation of medical student surgical knowledge correlate with written and oral exam performance? J Surg Res 104(1):36–39CrossRefPubMedGoogle Scholar
  3. 3.
    Martin J, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (osats) for surgical residents. Br J Surg 84(2):273–278CrossRefPubMedGoogle Scholar
  4. 4.
    Reznick R, MacRae H (2006) Teaching surgical skills-changes in the wind. N Engl J Med 355(25):2664CrossRefPubMedGoogle Scholar
  5. 5.
    Yu T, Wheeler B, Hill A (2011) Clinical supervisor evaluations during general surgery clerkships. Med Teach 33(9):479–484CrossRefGoogle Scholar
  6. 6.
    Datta V, Bann S, Mandalia M, Darzi A (2006) The surgical efficiency score: a feasible, reliable, and valid method of skills assessment. Am J Surg 192(3):372–378CrossRefPubMedGoogle Scholar
  7. 7.
    Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ Br Med J 327(7422):1032CrossRefGoogle Scholar
  8. 8.
    Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. arXiv preprint arXiv:1602.03012
  9. 9.
    Lea C, Hager GD, Vidal R (2015) An improved model for segmentation and recognition of fine-grained activities with application to surgical training tasks. In: 2015 IEEE winter conference on applications of computer vision, pp 1123–1129Google Scholar
  10. 10.
    Zia A, Sharma Y, Bettadapura V, Sarin EL, Clements MA, Essa I (2015) Automated assessment of surgical skills using frequency analysis. In: Medical image computing and computer-assisted intervention–MICCAI 2015. Springer, pp 430–438Google Scholar
  11. 11.
    Sharma Y, Plötz T, Hammerla N, Mellor S, Roisin M, Olivier P, Deshmukh S, McCaskie A, Essa I (2014) Automated surgical OSATS prediction from videos. In: ISBI, IEEEGoogle Scholar
  12. 12.
    Sharma Y, Bettadapura V, Plötz T, Hammerla N, Mellor S, McNaney R, Olivier P, Deshmukh S, McCaskie A, Essa I (2014) Video based assessment of OSATS using sequential motion textures. In: International workshop on modeling and monitoring of computer assisted interventions (M2CAI)-workshopGoogle Scholar
  13. 13.
    Tao L, Zappella L, Hager GD, Vidal R (2013) Surgical gesture segmentation and recognition. In: Medical image computing and computer-assisted intervention–MICCAI 2013. Springer, pp 339–346Google Scholar
  14. 14.
    Bettadapura V, Schindler G, Plötz T, Essa I (2013) Augmenting bag-of-words: data-driven discovery of temporal and structural information for activity recognition. In: IEEE CVPRGoogle Scholar
  15. 15.
    Haro BB, Zappella L, Vidal R (2012) Surgical gesture classification from video data. In: MICCAI 2012. Springer, pp 34–41Google Scholar
  16. 16.
    Zappella L, Béjar B, Hager G, Vidal R (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17(7):732–745CrossRefPubMedGoogle Scholar
  17. 17.
    Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641CrossRefPubMedGoogle Scholar
  18. 18.
    Lalys F, Riffaud L, Bouget D, Jannin P (2011) An application-dependent framework for the recognition of high-level surgical tasks in the or. In: Medical image computing and computer-assisted intervention—MICCAI 2011. Springer, pp 331–338Google Scholar
  19. 19.
    Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: Medical image computing and computer-assisted intervention—MICCAI 2010. Springer, pp 400–407Google Scholar
  20. 20.
    Lin H, Hager G (2009) User-independent models of manipulation using video contextual cues. In: International workshop on modeling and monitoring of computer assisted interventions (M2CAI)Google Scholar
  21. 21.
    Judkins TN, Oleynikov D, Stergiou N (2009) Objective evaluation of expert and novice performance during robotic surgical training tasks. Surg Endosc 23(3):590–597CrossRefPubMedGoogle Scholar
  22. 22.
    Pirsiavash H, Vondrick C, Torralba A (2014) Assessing the quality of actions. In: ECCV. Springer, pp 556–571Google Scholar
  23. 23.
    Wang H, Kläser A, Schmid C, Liu CL (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79CrossRefGoogle Scholar
  24. 24.
    Liu J, Kuipers B, Savarese S (2011) Recognizing human actions by attributes. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 3337–3344Google Scholar
  25. 25.
    Niebles JC, Chen CW, Fei-Fei L (2010) Modeling temporal structure of decomposable motion segments for activity classification. In: Computer vision–ECCV 2010. Springer, pp 392–405Google Scholar
  26. 26.
    Laptev I, Lindeberg T (2003) Space-time interest points. In: IN ICCV, pp 432–439Google Scholar
  27. 27.
    Wang H, Ullah MM, Kläser A, Laptev I, Schmid C (2009) Evaluation of local spatio-temporal features for action recognition. In: BMVCGoogle Scholar
  28. 28.
    Reiley C, Lin H, Yuh D, Hager G (2011) Review of methods for objective surgical skill evaluation. Surg Endosc 25(2):356–366CrossRefPubMedGoogle Scholar
  29. 29.
    Reiley CE, Hager GD (2009) Decomposition of robotic surgical tasks: an analysis of subtasks and their correlation to skill. In: M2CAI workshop. MICCAI, LondonGoogle Scholar

Copyright information

© CARS 2016

Authors and Affiliations

  • Aneeq Zia
    • 1
    Email author
  • Yachna Sharma
    • 1
  • Vinay Bettadapura
    • 1
  • Eric L. Sarin
    • 2
  • Thomas Ploetz
    • 3
  • Mark A. Clements
    • 1
  • Irfan Essa
    • 1
  1. 1.Georgia Tech College of ComputingAtlantaUSA
  2. 2.Department of SurgeryEmory UniversityAtlantaUSA
  3. 3.School of Computing Science, Newcastle UniversityNewcastleUK

Personalised recommendations