Objective assessment of psychomotor skills has become an important challenge in the training of minimally invasive surgical (MIS) techniques. Currently, no gold standard defining surgical competence exists for classifying residents according to their surgical skills. Supervised classification has been proposed as a means for objectively establishing competence thresholds in psychomotor skills evaluation. This report presents a study comparing three classification methods for establishing their validity in a set of tasks for basic skills’ assessment.
Linear discriminant analysis (LDA), support vector machines (SVM), and adaptive neuro-fuzzy inference systems (ANFIS) were used. A total of 42 participants, divided into an experienced group (4 expert surgeons and 14 residents with >10 laparoscopic surgeries performed) and a nonexperienced group (16 students and 8 residents with <10 laparoscopic surgeries performed), performed three box trainer tasks validated for assessment of MIS psychomotor skills. Instrument movements were captured using the TrEndo tracking system, and nine motion analysis parameters (MAPs) were analyzed. The performance of the classifiers was measured by leave-one-out cross-validation using the scores obtained by the participants.
The mean accuracy performances of the classifiers were 71 % (LDA), 78.2 % (SVM), and 71.7 % (ANFIS). No statistically significant differences in the performance were identified between the classifiers.
The three proposed classifiers showed good performance in the discrimination of skills, especially when information from all MAPs and tasks combined were considered. A correlation between the surgeons’ previous experience and their execution of the tasks could be ascertained from results. However, misclassifications across all the classifiers could imply the existence of other factors influencing psychomotor competence.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Aggarwal R, Moorthy K, Darzi A (2004) Laparoscopic skills training and assessment. Br J Surg 91:1549–1558
Usón J, Sánchez-Margallo FM, Pascual S, Climent S (2010) Formación en cirugía laparoscópica paso a paso, 4th edn. Centro de Cirugía de Mínima Invasión Jesús Usón, Cáceres
Fried GM, Feldman LS (2008) Objective assessment of technical performance. World J Surg 32:156–160. doi:10.1007/s00268-007-9143-y
Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. Br Med J 327:1032–1037
Feldman LS, Sherman V, Fried GM (2004) Using simulators to assess laparoscopic competence: ready for widespread use? Surgery 135:28–42. doi:10.1016/S0039-6060(03)00155-7
Hiemstra E (2012) Acquiring minimally invasive surgical skills. Department of Minimally Invasive Surgery in Gynaecology, Faculty of Medicine/Leiden University Medical Center (LUMC), Leiden University, Leiden
Ritter EM, Scott DJ (2007) Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 14:107–112. doi:10.1177/1553350607302329
Oropesa I, Sanchez-Gonzalez P, Lamata P, Chmarra MK, Pagador JB, Sanchez-Margallo JA, Sanchez-Margallo FM, Gomez EJ (2011) Methods and tools for objective assessment of psychomotor skills in laparoscopic surgery. J Surg Res 171:E81–E95. doi:10.1016/j.jss.2011.06.034
Chmarra MK, Klein S, de Winter JCF, Jansen FW, Dankelman J (2010) Objective classification of residents based on their psychomotor laparoscopic skills. Surg Endosc 24:1031–1039
Cotin S, Stylopoulos N, Ottensmeyer M, Neumann P, Rattner D, Dawson S (2002) Metrics for laparoscopic skills trainers: the weakest link! In: Dohi T, Kikinis R (eds) Medical image computing and computer-assisted intervention—MICCAI 2002. Springer, Berlin, pp 35–43
Rosen J, Solazzo M, Hannaford B, Sinanan M (2002) Task decomposition of laparoscopic surgery for objective evaluation of surgical residents’ learning curve using hidden Markov model. Comput Aided Surg 7:49–61
Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, Fried GM (2003) Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 17:964–967. doi:10.1007/s00464-002-8828-4
Feldman LS, Hagarty SE, Ghitulescu G, Stanbridge D, Fried GM (2004) Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg 198:105–110
Chmarra MK, Grimbergen CA, Jansen FW, Dankelman J (2010) How to objectively classify residents based on their psychomotor laparoscopic skills? Minim Invasive Ther Allied Technol 19:2–11. doi:10.3109/13645700903492977
Reiley CE, Hager GD (2009) Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Med Image Comput Comput Assist Interv 12:435–442
Leong JJH, Nicolaou M, Atallah L, Mylonas GP, Darzi A (2007) HMM assessment of quality of movement trajectory in laparoscopic surgery. Comput Aided Surg 12:335–346
Megali G, Sinigaglia S, Tonet O, Dario P (2006) Modelling and evaluation of surgical performance using hidden Markov models. IEEE Trans Biomed Eng 53:1911–1919. doi:10.1109/TBME.2006.881784
Allen BF, Nistor V, Dutson E, Carman G, Lewis C, Faloutsos P (2010) Support vector machines improve the accuracy of evaluation for the performance of laparoscopic training tasks. Surg Endosc 24:170–178. doi:10.1007/s00464-009-0556-6
Riojas M, Feng C, Hamilton A, Rozenblit J (2011) Knowledge elicitation for performance assessment in a computerized surgical training system. Appl Soft Comput 11:3697–3708. doi:10.1016/j.asoc.2011.01.041
Huang J, Payandeh S, Doris P, Hajshirmohammadi I (2005) Fuzzy classification: towards evaluating performance on a surgical simulator. Stud Health Technol Inform 111:194–200
Oropesa I, Chmarra MK, Sanchez-Gonzalez P, Lamata P, Rodrigues SP, Enciso S, Sanchez-Margallo FM, Jansen FW, Dankelman J, Gomez EJ (2012) Relevance of motion-related assessment metrics in laparoscopic surgery. Surg Innov 20:302–315. doi:10.1177/1553350612459808
Duda RO, Hart PE, Stork DG (2001) Pattern classification, 2nd edn. Wiley, New York
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297
Jang J-R (1993) ANFIS: adaptive-network-based fuzzy inference system: systems, man, and cybernetics. IEEE Trans 23:665–685
Pérez-Rodríguez R, Marcano-Cedeño A, Costa Ú, Solana J, Cáceres C, Opisso E, Tormos JM, Medina J, Gómez EJ (2012) Inverse kinematics of a 6 DoF human upper limb using ANFIS and ANN for anticipatory actuation in ADL-based physical neurorehabilitation. Expert Syst Appl 39:9612–9622. doi:10.1016/j.eswa.2012.02.143
Chang C, Lin C (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27. doi:10.1145/1961189.1961199
Jain AK, Duin RPW, Mao Jianchang (2000) Statistical pattern recognition: a review: pattern analysis and machine intelligence. IEEE Trans 22:4–37
Darzi A, Datta V, Mackay S (2001) The challenge of objective assessment of surgical skill. Am J Surg 181:484–486
Horeman T, Rodrigues SP, van den Dobbelsteen JJ, Jansen FW, Dankelman J (2012) Visual force feedback in laparoscopic training. Surg Endosc 26:242–248. doi:10.1007/s00464-011-1861-4
Soreide K, Glomsaker T, Soreide JA (2008) Surgery in Norway: beyond the scalpel in the 21st century. Arch Surg 143:1011–1016. doi:10.1001/archsurg.143.10.1011
Dumon KR, Traynor O, Broos P, Gruwez JA, Darzi A, Williams NN (2004) Surgical education in the new millennium: the European perspective. Surg Clin North Am 84:1471–1491, viii. doi:10.1016/j.suc.2004.06.005
Aggarwal R, Grantcharov T, Moorthy K, Milland T, Papasavas P, Dosis A, Bello F, Darzi A (2007) An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room. Ann Surg 245:992–999
Yamaguchi S, Konishi K, Yasunaga T, Yoshida D, Kinjo N, Kobayashi K, Ieiri S, Okazaki K, Nakashima H, Tanoue K (2007) Construct validity for eye–hand coordination skill on a virtual reality laparoscopic surgical simulator. Surg Endosc 21:2253–2257
The authors thank all the surgeons, residents, and medical students who kindly volunteered and participated in the clinical trials, as well as the staff of the skills laboratory at the Leiden University Medical Centre for providing the available working space.
Ignacio Oropesa, Patricia Sánchez-González, Frank Willem Jansen, Jenny Dankelman, and Enrique J. Gómez participate under partial funding of LLP-Leonardo da Vinci project MISTELA (528125-LLP-1-2012-1-UK). Magdalena K. Chmarra, Pablo Lamata, and Rodrigo Pérez-Rodríguez have no conflicts of interest or financial ties to disclose.
Rights and permissions
About this article
Cite this article
Oropesa, I., Sánchez-González, P., Chmarra, M.K. et al. Supervised classification of psychomotor competence in minimally invasive surgery based on instruments motion analysis. Surg Endosc 28, 657–670 (2014). https://doi.org/10.1007/s00464-013-3226-7
- Minimally invasive surgery
- Objective assessment
- Supervised classification