Surgical Endoscopy

, Volume 25, Issue 2, pp 356–366 | Cite as

Review of methods for objective surgical skill evaluation

  • Carol E. Reiley
  • Henry C. Lin
  • David D. Yuh
  • Gregory D. Hager



Rising health and financial costs associated with iatrogenic errors have drawn increasing attention to the dexterity of surgeons. With the advent of new technologies, such as robotic surgical systems and medical simulators, researchers now have the tools to analyze surgical motion with the goal of differentiating the level of technical skill in surgeons.


The review for this paper is obtained from a Google Scholar and PubMed search of the key words “objective surgical skill evaluation.” Only studies that included motion analysis were used.


In this paper, we provide a clinical motivation for the importance of surgical skill evaluation. We review the current methods of tracking surgical motion and the available data-collection systems. We also survey current methods of surgical skill evaluation and show that most approaches fall into one of three methods: (1) structured human grading; (2) descriptive statistics; or (3) statistical language models of surgical motion. We discuss the need for an encompassing approach to model human skill through statistical models to allow for objective skill evaluation.


Surgical, training Courses, human Robotic, training 


  1. 1.
    Reznick RK, MacRae H (2006) Teaching surgical skills-changes in the wind. N Engl J Med 355(25):2664–2669CrossRefPubMedGoogle Scholar
  2. 2.
    DeFrances CJ, Hall MJ (2005) National hospital discharge survey. Adv Data Vital Health Stat 385:1–19Google Scholar
  3. 3.
    Anderson P, Gelijns A, Moskowitz A, Arons R, Gupta L, Weinberg A, Faries P, Nowygrod R, Kent K (2004) Understanding trends in inpatient surgical volume: vascular interventions, 1980–2000. J Vasc Surg 39(6):1200–1208CrossRefPubMedGoogle Scholar
  4. 4.
    Zhan C, Miller M (2003) Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA 290(14):1868–1874CrossRefPubMedGoogle Scholar
  5. 5.
    Peracchia A (2001) Surgical education in the third millennium. Ann Surg 234(6):709–712CrossRefPubMedGoogle Scholar
  6. 6.
    Carter BN (1952) The fruition of Halsted’s concept of surgical training. Surgery 32:518–527PubMedGoogle Scholar
  7. 7.
    Darzi A, Smith S, Taffinder N (1999) Assessing operative skill needs to become more objective. Br Med J 318:887–888Google Scholar
  8. 8.
    Barden CB, Specht MC, McCarter MD, Daly JM, Fahey TJ (2003) Effects of limited work hours on surgical training. Obstet Gynecol Surv 58(4):244–245CrossRefGoogle Scholar
  9. 9.
    Ericsson KA (2004) Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domain. Acad Med 79:70–81CrossRefGoogle Scholar
  10. 10.
    Bann SD, Datta VK, Khan MS, Ridgway PF, Darzi AW (2005) Attitudes towards skills examinations for basic surgical trainees. Int J Clin Pract 59(1):107–113CrossRefPubMedGoogle Scholar
  11. 11.
    Kopta JA (1971) An approach to the evaluation of operative skills. Surgery 70:297–303PubMedGoogle Scholar
  12. 12.
    Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165:358–361CrossRefPubMedGoogle Scholar
  13. 13.
    Lin HC, Shafran I, Yuh DD, Hager GD (2006) Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comp Aid Surg 11(5):220–223CrossRefGoogle Scholar
  14. 14.
    Bridgewater B, Grayson AD, Jackson M, Brooks N, Grotte GJ, Keenan DJM, Millner R, Fabri BM, Jones M (2003) Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 327:13–17CrossRefPubMedGoogle Scholar
  15. 15.
    Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. Br Med Assoc 327:1032–1037Google Scholar
  16. 16.
    Reznick R, Regehr G, MacRae H, Martin J, McCulloch W (1997) Testing technical skill via an innovative bench station examination. Am J Surg 173(3):226–230CrossRefPubMedGoogle Scholar
  17. 17.
    Doyle J, Webber E, Sidhu R (2007) A universal global rating scale for the evaluation of technical skills in the operating room. Am J Surg 193(5):551–555CrossRefPubMedGoogle Scholar
  18. 18.
    Neumann M, Hahn C, Horbach T, Schneide I, Meining A, Heldwein W, Rsch T (2003) Score card endoscopy: a multicenter study to evaluate learning curves in 1-week courses using the Erlangen endotrainer. Endoscopy 35:515–520CrossRefPubMedGoogle Scholar
  19. 19.
    Moorthy K, Munz Y, Dosis A, Hernandez J, Martin S, Bello F, Rockall T, Darzi A (2004) Dexterity enhancement with robotic surgery. Surg Endosc 18:790–795PubMedGoogle Scholar
  20. 20.
    Shah J, Darzi A (2001) Surgical skills assessment: an ongoing debate. BJU Int 88(7):655–660CrossRefPubMedGoogle Scholar
  21. 21.
    Rosenbaum DA (1992) Reaching and grasping. Human Motor Control 197–225Google Scholar
  22. 22.
    Chmarra MK, Grimbergen CA, Dankelman J (2007) Systems for tracking minimally invasive surgical instruments. Min Inv Ther All Tech 16(6):328–340CrossRefGoogle Scholar
  23. 23.
    Riedl S (2002) Modern operating room management in the workflow of surgery, spectrum of tasks and challenges of the future. Anaesthesist 73:105–110Google Scholar
  24. 24.
    Thompson CA (2004) Jcaho gears up to survey sterile compounding practices. Am J Health, Syst PharmGoogle Scholar
  25. 25.
    Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF, Schirmer B, Hoffman K (2004) Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery 135:21–27CrossRefPubMedGoogle Scholar
  26. 26.
    Figert PL, Park AE, Witzke DB, Schwartz RW (2001) Transfer of training in acquiring laparoscopic skills. J Am Coll Surg 193(5):533–537CrossRefPubMedGoogle Scholar
  27. 27.
    Moltenbrey K (2006) Tools of the trade. Comp Graph World 29(11)Google Scholar
  28. 28.
    5DT Data Glove UltaGoogle Scholar
  29. 29.
    Francis N (2002) The performance of master surgeons on the advanced Dundee endoscopic psychomotor tester. Arch Surg 137:841–844CrossRefPubMedGoogle Scholar
  30. 30.
    Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in laboratory-based model. J Am Coll Surg 193:479–485CrossRefPubMedGoogle Scholar
  31. 31.
    Taffinder N, Smith S, Mair J, Russel RC, Darzi A (1999) Can a computer measure surgical precision? Reliability, validity and feasibility of the ICSAD. Surg Endosc 13(1):81CrossRefGoogle Scholar
  32. 32.
    Xin H, Zelek JS, Carnahan H (2006) Laparoscopic surgery, perceptual limitations and force: a review. In: First joint eurohaptics conference and symposium on haptic interfaces for virtual environmentGoogle Scholar
  33. 33.
    Guthart GS, Salisbury JK (2000) The intuitive telesurgery system: overview and application. In: IEEE International conference on robotics automation, pp 618–621Google Scholar
  34. 34.
    Intuitive Surgical Inc (2007) Intuitive surgical annual reportGoogle Scholar
  35. 35.
    Nawrat Z, Kostka P (2007) The robin heart vision, telemanipulator for camera holding––preliminary test results. J Automat Mobile Robotics Intelligent Syst 1(1):48–53Google Scholar
  36. 36.
    Mayer H, Gomex F, Wierstra D, Nagy I, Knoll A, Schmidhuber J (2006) A system for robotic heart surgery that learns to tie knots using recurrent neural networks. ICRA, pp 543–548Google Scholar
  37. 37.
    Global Strategic Business (2007) Computer assisted surgical systems. A global strategic business reportGoogle Scholar
  38. 38.
    Richards C, Rosen J, Hannaford B, Pellegrini C, Sinanan M (2000) Skills evaluation in minimally invasive surgery using force/torque signatures. Surg Endosc 14:791–798CrossRefPubMedGoogle Scholar
  39. 39.
    Rosen J, Brown JD, Chang L, Sinanan MN, Hannaford B (2006) Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Trans Biomed Eng 53(3):399–413CrossRefPubMedGoogle Scholar
  40. 40.
    Pugh CM, Youngblood P (2002) Development and validation of assessment measures for a newly developed physical examination simulator. J Am Med Inform Assoc 9(5):448–460CrossRefPubMedGoogle Scholar
  41. 41.
    Ko S, Kwon D (2004) A surgical knowledge based interaction method for a laparoscopic assistant robot. In: RO-MAN––IEEE, International workshop on robot and human interactive communication, pp 313–318Google Scholar
  42. 42.
    Ibbotson JA, MacKenzie CL, Cao CG, Lomax AJ (1999) Gaze patterns in laparoscopic surgery. In: MMVR, pp 154–160Google Scholar
  43. 43.
    Rosen J, Hannaford B, Richards CG, Sinanan MN (2001) Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 48(5):579–591CrossRefPubMedGoogle Scholar
  44. 44.
    Yamauchi Y, Yamashita J, Morikawa O, Hashimoto R (2002) Surgical skill evaluation by force data for endoscopic sinus surgery training system. In: MICCAI, pp 44–51Google Scholar
  45. 45.
    Judkins TN, Oleynikov D, Stergiou N (2008) Objective evaluation of expert performance during human robotic surgical procedures. J Robotics Surg 1(4):307–312CrossRefGoogle Scholar
  46. 46.
    Van Sickle KR, McClusky DA, Gallagher AG, Smith CD (2005) Construct validation of the promise simulator using a novel laparoscopic suturing task. Surg Endosc 19(9):1227–1231CrossRefPubMedGoogle Scholar
  47. 47.
    Acosta E, Temkin B (2005) Haptic laparoscopic skills trainer with practical user evaluation metrics. MMVR 111:8–11Google Scholar
  48. 48.
    Hernandez JD, Bann SD, Munz Y, Moorthy K, Datta V (2004) Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surg Endosc 372–378Google Scholar
  49. 49.
    Rosen J, Solazzo M, Hannaford B, Sinanan MN (2002) Task decomposition of laparoscopic surgery for objective evaluation of surgical residents’ learning curve using hidden Markov model. Comp Aid Surg 7(1):49–61CrossRefGoogle Scholar
  50. 50.
    Cao CG, MacKenzie RD, Payandeh S (1996) Task and motion analyses in endoscopic surgery. In: ASME IMECE, pp 583–590Google Scholar
  51. 51.
    Padoy N, Blum T, Essa IA, Feussne H, Berger MO, Navab N (2007) A boosted segmentation method for surgical workflow analysis. In: MICCAI, pp 102–109Google Scholar
  52. 52.
    Datta V, Bann S, Hernandez J, Darzi A (2007) Objective assessment comparing hand-assisted and conventional laparoscopic surgery. Surg Endosc 21:414–417CrossRefPubMedGoogle Scholar
  53. 53.
    Datta V, Mandalia M, Mackay S, Chang A, Cheshire N, Darzi A (2002) Relationship between skill and outcome in the laboratory based model. Surgery 131:318–323CrossRefPubMedGoogle Scholar
  54. 54.
    Fried GM, Feldman LS, Vassiliou MC, Stanbridge D, Ghitulescu G, Andrew CG (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–528CrossRefPubMedGoogle Scholar
  55. 55.
    Acosta E, Temkin B (2005) Dynamic generation of surgery specific simulators: a feasibility study. StudHealth Technol Inform 111:1–7Google Scholar
  56. 56.
    Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J (2002) Assessment of technical surgical skills. Eur J Surg 168:139–144CrossRefPubMedGoogle Scholar
  57. 57.
    Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A (2004) Laparoscopic virtual reality and box trainers: is one superior to the other? Surg Endosc 18:485–494CrossRefPubMedGoogle Scholar
  58. 58.
    Cotin S, Stylopoulos N, Ottensmeyer M, Neumann P, Rattner D, Dawson S (2002) Metrics for laparoscopic skills trainers: the weakest link. MICCAI 2488:35–43Google Scholar
  59. 59.
    O’Toole RV, Playter RR, Krummel TM, Blank WC, Cornelius NH, Roberts WR, Bell WJ, Raibert M (1999) Measuring and developing suturing technique with a VR surgical simulator. Am Coll Surg 189(1):114–127CrossRefGoogle Scholar
  60. 60.
    Liu A, Tendick F, Cleary K, Kaufmann C (2003) A survey of surgical simulation: applications, technology, and education. Presence Teleop Virt Environ 12(6):599–614CrossRefGoogle Scholar
  61. 61.
    Issenberg SB, McGaghie WC (2002) Clinical skills training practice makes perfect. Med Educat 36(3):210–211CrossRefGoogle Scholar
  62. 62.
    Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM (2005) Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 241(2):364–372CrossRefPubMedGoogle Scholar
  63. 63.
    Seymour NE, Anthony G, Sanziana A, O’Brien MK, Bansal VK, Andersen DK (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4):458–464CrossRefPubMedGoogle Scholar
  64. 64.
    Chaudhry A, Sutton C, Wood J, Stone R, McCloy R (1999) Learning rate for laparoscopic surgical skills on mist VR, a virtual reality simulator: quality of human-computer interface. Ann R Coll Surg Engl 81:281286Google Scholar
  65. 65.
    Nagy I, Mayer H, Knoll A (2003) The Endopar system for minimally invasive robotic surgery. Technical report, TUMGoogle Scholar
  66. 66.
    MacKenzie CL, Ibbotson JA, Cao CLGL, Lomax AJ (2001) Hierarchical decomposition of laparoscopic surgery: a human factors approach to investigating the operating room environment. Minim Invas Ther Allied Technol 10(3):121–127CrossRefGoogle Scholar
  67. 67.
    Ahmadi SA, Sielhorst T, Stauder R, Horn M, Feussner H, Navab N (2006) Recovery of surgical workflow without explicit models. Med Imag Comput Comput Assist Interv Int Conf Med Image Comput Comput Assist Interv 9(1):420–428Google Scholar
  68. 68.
    Mackel T, Rosen J, Pugh C (2007) Application of hidden Markov modeling to objective medical skill evaluation. Stud Health Technol Inform 125:316–318PubMedGoogle Scholar
  69. 69.
    Leong JH, Nicolaou M, Atallah L, Mylonas GP, Darzi AW, Yang G (2006) Hmm assessment of quality of movement trajectory in laparoscopic surgery. MICCAI 4190:752–759Google Scholar
  70. 70.
    Speidel S, Delles ML, Gutt C, Dillmann R (2006) Tracking of instruments in minimally invasive surgery for surgical skill analysis. Med Imag Augment Real 4091:148–155CrossRefGoogle Scholar
  71. 71.
    Hundtofte CS, Hager GD, Okamura AM (2002) Building a task language for segmentation and recognition of user input to cooperative manipulation systems. Haptic Interf Virt Environ Teleop Syst, pp 225–230Google Scholar
  72. 72.
    Murphy TE, Vignes CM, Yuh DD, Okamura AM (2003) Automatic motion recognition and skill evaluation for dynamic tasks. Eurohaptics 363–373Google Scholar
  73. 73.
    Reiley CE, Lin HC, Varadarajan B, Khudanpur S, Yuh DD, Hager GD (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Med Virt Real 132:396–401Google Scholar
  74. 74.
    Castellani A, Botturi D, Bicego M, Fiorini P (2004) Hybrid hmm/svm model for the analysis and segmentation of teleoperation tasks. IEEE Int Conf Robotics Autom 3:2918–2923Google Scholar
  75. 75.
    Li M, Okamura AM (2003) Recognition of operator motions for realtime assistance using virtual fixtures. Int Sympos Haptic Interf Virt Environ Teleop Syst 1954–1959Google Scholar
  76. 76.
    Kragic D, Marayong P, Li M, Okamura AM, Hager GD (2005) Human-machine collaborative systems for microsurgical applications. Int J Robotics Res 24(9):731–741CrossRefGoogle Scholar
  77. 77.
  78. 78.
    Okamura AM (2004) Methods for haptic feedback in teleoperated robot-assisted surgery. Industr Robot Int J 31:499–508CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Carol E. Reiley
    • 1
  • Henry C. Lin
    • 1
  • David D. Yuh
    • 2
    • 3
  • Gregory D. Hager
    • 1
  1. 1.Laboratory of Computational Sensing and Robotics Department of Computer ScienceJohns Hopkins UniversityBaltimoreUSA
  2. 2.Program for Outcomes Research, Department of SurgeryJohns Hopkins School of MedicineBaltimoreUSA
  3. 3.Division of Cardiac SurgeryJohns Hopkins Medical InstitutionsBaltimoreUSA

Personalised recommendations