Skip to main content

Supervised classification of psychomotor competence in minimally invasive surgery based on instruments motion analysis

Surgical Endoscopy volume 28, pages 657–670 (2014)Cite this article

Abstract

Background

Objective assessment of psychomotor skills has become an important challenge in the training of minimally invasive surgical (MIS) techniques. Currently, no gold standard defining surgical competence exists for classifying residents according to their surgical skills. Supervised classification has been proposed as a means for objectively establishing competence thresholds in psychomotor skills evaluation. This report presents a study comparing three classification methods for establishing their validity in a set of tasks for basic skills’ assessment.

Methods

Linear discriminant analysis (LDA), support vector machines (SVM), and adaptive neuro-fuzzy inference systems (ANFIS) were used. A total of 42 participants, divided into an experienced group (4 expert surgeons and 14 residents with >10 laparoscopic surgeries performed) and a nonexperienced group (16 students and 8 residents with <10 laparoscopic surgeries performed), performed three box trainer tasks validated for assessment of MIS psychomotor skills. Instrument movements were captured using the TrEndo tracking system, and nine motion analysis parameters (MAPs) were analyzed. The performance of the classifiers was measured by leave-one-out cross-validation using the scores obtained by the participants.

Results

The mean accuracy performances of the classifiers were 71 % (LDA), 78.2 % (SVM), and 71.7 % (ANFIS). No statistically significant differences in the performance were identified between the classifiers.

Conclusions

The three proposed classifiers showed good performance in the discrimination of skills, especially when information from all MAPs and tasks combined were considered. A correlation between the surgeons’ previous experience and their execution of the tasks could be ascertained from results. However, misclassifications across all the classifiers could imply the existence of other factors influencing psychomotor competence.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Aggarwal R, Moorthy K, Darzi A (2004) Laparoscopic skills training and assessment. Br J Surg 91:1549–1558

    Article  CAS  PubMed  Google Scholar 

  2. Usón J, Sánchez-Margallo FM, Pascual S, Climent S (2010) Formación en cirugía laparoscópica paso a paso, 4th edn. Centro de Cirugía de Mínima Invasión Jesús Usón, Cáceres

    Google Scholar 

  3. Fried GM, Feldman LS (2008) Objective assessment of technical performance. World J Surg 32:156–160. doi:10.1007/s00268-007-9143-y

    Article  PubMed  Google Scholar 

  4. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. Br Med J 327:1032–1037

    Article  Google Scholar 

  5. Feldman LS, Sherman V, Fried GM (2004) Using simulators to assess laparoscopic competence: ready for widespread use? Surgery 135:28–42. doi:10.1016/S0039-6060(03)00155-7

    Article  PubMed  Google Scholar 

  6. Hiemstra E (2012) Acquiring minimally invasive surgical skills. Department of Minimally Invasive Surgery in Gynaecology, Faculty of Medicine/Leiden University Medical Center (LUMC), Leiden University, Leiden

  7. Ritter EM, Scott DJ (2007) Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 14:107–112. doi:10.1177/1553350607302329

    Article  PubMed  Google Scholar 

  8. Oropesa I, Sanchez-Gonzalez P, Lamata P, Chmarra MK, Pagador JB, Sanchez-Margallo JA, Sanchez-Margallo FM, Gomez EJ (2011) Methods and tools for objective assessment of psychomotor skills in laparoscopic surgery. J Surg Res 171:E81–E95. doi:10.1016/j.jss.2011.06.034

    Article  PubMed  Google Scholar 

  9. Chmarra MK, Klein S, de Winter JCF, Jansen FW, Dankelman J (2010) Objective classification of residents based on their psychomotor laparoscopic skills. Surg Endosc 24:1031–1039

    Article  PubMed Central  PubMed  Google Scholar 

  10. Cotin S, Stylopoulos N, Ottensmeyer M, Neumann P, Rattner D, Dawson S (2002) Metrics for laparoscopic skills trainers: the weakest link! In: Dohi T, Kikinis R (eds) Medical image computing and computer-assisted intervention—MICCAI 2002. Springer, Berlin, pp 35–43

    Chapter  Google Scholar 

  11. Rosen J, Solazzo M, Hannaford B, Sinanan M (2002) Task decomposition of laparoscopic surgery for objective evaluation of surgical residents’ learning curve using hidden Markov model. Comput Aided Surg 7:49–61

    Article  PubMed  Google Scholar 

  12. Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, Fried GM (2003) Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 17:964–967. doi:10.1007/s00464-002-8828-4

    Article  CAS  PubMed  Google Scholar 

  13. Feldman LS, Hagarty SE, Ghitulescu G, Stanbridge D, Fried GM (2004) Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg 198:105–110

    Article  PubMed  Google Scholar 

  14. Chmarra MK, Grimbergen CA, Jansen FW, Dankelman J (2010) How to objectively classify residents based on their psychomotor laparoscopic skills? Minim Invasive Ther Allied Technol 19:2–11. doi:10.3109/13645700903492977

    Article  PubMed  Google Scholar 

  15. Reiley CE, Hager GD (2009) Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Med Image Comput Comput Assist Interv 12:435–442

    PubMed  Google Scholar 

  16. Leong JJH, Nicolaou M, Atallah L, Mylonas GP, Darzi A (2007) HMM assessment of quality of movement trajectory in laparoscopic surgery. Comput Aided Surg 12:335–346

    PubMed  Google Scholar 

  17. Megali G, Sinigaglia S, Tonet O, Dario P (2006) Modelling and evaluation of surgical performance using hidden Markov models. IEEE Trans Biomed Eng 53:1911–1919. doi:10.1109/TBME.2006.881784

    Article  PubMed  Google Scholar 

  18. Allen BF, Nistor V, Dutson E, Carman G, Lewis C, Faloutsos P (2010) Support vector machines improve the accuracy of evaluation for the performance of laparoscopic training tasks. Surg Endosc 24:170–178. doi:10.1007/s00464-009-0556-6

    Article  PubMed  Google Scholar 

  19. Riojas M, Feng C, Hamilton A, Rozenblit J (2011) Knowledge elicitation for performance assessment in a computerized surgical training system. Appl Soft Comput 11:3697–3708. doi:10.1016/j.asoc.2011.01.041

    Article  Google Scholar 

  20. Huang J, Payandeh S, Doris P, Hajshirmohammadi I (2005) Fuzzy classification: towards evaluating performance on a surgical simulator. Stud Health Technol Inform 111:194–200

    PubMed  Google Scholar 

  21. Oropesa I, Chmarra MK, Sanchez-Gonzalez P, Lamata P, Rodrigues SP, Enciso S, Sanchez-Margallo FM, Jansen FW, Dankelman J, Gomez EJ (2012) Relevance of motion-related assessment metrics in laparoscopic surgery. Surg Innov 20:302–315. doi:10.1177/1553350612459808

    Google Scholar 

  22. Duda RO, Hart PE, Stork DG (2001) Pattern classification, 2nd edn. Wiley, New York

    Google Scholar 

  23. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297

    Google Scholar 

  24. Jang J-R (1993) ANFIS: adaptive-network-based fuzzy inference system: systems, man, and cybernetics. IEEE Trans 23:665–685

    Google Scholar 

  25. Pérez-Rodríguez R, Marcano-Cedeño A, Costa Ú, Solana J, Cáceres C, Opisso E, Tormos JM, Medina J, Gómez EJ (2012) Inverse kinematics of a 6 DoF human upper limb using ANFIS and ANN for anticipatory actuation in ADL-based physical neurorehabilitation. Expert Syst Appl 39:9612–9622. doi:10.1016/j.eswa.2012.02.143

    Article  Google Scholar 

  26. Chang C, Lin C (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27. doi:10.1145/1961189.1961199

    Article  Google Scholar 

  27. Jain AK, Duin RPW, Mao Jianchang (2000) Statistical pattern recognition: a review: pattern analysis and machine intelligence. IEEE Trans 22:4–37

    Google Scholar 

  28. Darzi A, Datta V, Mackay S (2001) The challenge of objective assessment of surgical skill. Am J Surg 181:484–486

    Article  CAS  PubMed  Google Scholar 

  29. Horeman T, Rodrigues SP, van den Dobbelsteen JJ, Jansen FW, Dankelman J (2012) Visual force feedback in laparoscopic training. Surg Endosc 26:242–248. doi:10.1007/s00464-011-1861-4

    Article  PubMed Central  PubMed  Google Scholar 

  30. Soreide K, Glomsaker T, Soreide JA (2008) Surgery in Norway: beyond the scalpel in the 21st century. Arch Surg 143:1011–1016. doi:10.1001/archsurg.143.10.1011

    Article  PubMed  Google Scholar 

  31. Dumon KR, Traynor O, Broos P, Gruwez JA, Darzi A, Williams NN (2004) Surgical education in the new millennium: the European perspective. Surg Clin North Am 84:1471–1491, viii. doi:10.1016/j.suc.2004.06.005

    Google Scholar 

  32. Aggarwal R, Grantcharov T, Moorthy K, Milland T, Papasavas P, Dosis A, Bello F, Darzi A (2007) An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room. Ann Surg 245:992–999

    Article  PubMed Central  PubMed  Google Scholar 

  33. Yamaguchi S, Konishi K, Yasunaga T, Yoshida D, Kinjo N, Kobayashi K, Ieiri S, Okazaki K, Nakashima H, Tanoue K (2007) Construct validity for eye–hand coordination skill on a virtual reality laparoscopic surgical simulator. Surg Endosc 21:2253–2257

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The authors thank all the surgeons, residents, and medical students who kindly volunteered and participated in the clinical trials, as well as the staff of the skills laboratory at the Leiden University Medical Centre for providing the available working space.

Disclosures

Ignacio Oropesa, Patricia Sánchez-González, Frank Willem Jansen, Jenny Dankelman, and Enrique J. Gómez participate under partial funding of LLP-Leonardo da Vinci project MISTELA (528125-LLP-1-2012-1-UK). Magdalena K. Chmarra, Pablo Lamata, and Rodrigo Pérez-Rodríguez have no conflicts of interest or financial ties to disclose.

Author information

Authors and Affiliations

  1. Bioengineering and Telemedicine Centre (GBT), ETSI Telecomunicación, Universidad Politécnica de Madrid (UPM), Avda Complutense, 30, 28040, Madrid, Spain

    Ignacio Oropesa, Patricia Sánchez-González, Pablo Lamata, Rodrigo Pérez-Rodríguez & Enrique J. Gómez

  2. Networking Research Center on Bioengineering, Biomaterials and Nanomedicine (CIBER-BBN), Poeta Mariano Esquillor s/n, 50018, Zaragoza, Spain

    Ignacio Oropesa, Patricia Sánchez-González, Rodrigo Pérez-Rodríguez & Enrique J. Gómez

  3. Department of Circulation and Medical Imaging, Faculty of Medicine, Norwegian University of Science and Technology (NTNU), Trondheim, Norway

    Magdalena K. Chmarra

  4. Department of BioMechanical Engineering, Faculty of Mechanical, Maritime and Materials Engineering (3mE), Delft University of Technology, Mekelweg 2, 2628 CD, Delft, The Netherlands

    Frank Willem Jansen & Jenny Dankelman

  5. Department of Gynecology, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden, The Netherlands

    Frank Willem Jansen

Authors
  1. Ignacio Oropesa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patricia Sánchez-González
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Magdalena K. Chmarra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pablo Lamata
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rodrigo Pérez-Rodríguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Frank Willem Jansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jenny Dankelman
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Enrique J. Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ignacio Oropesa.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Oropesa, I., Sánchez-González, P., Chmarra, M.K. et al. Supervised classification of psychomotor competence in minimally invasive surgery based on instruments motion analysis. Surg Endosc 28, 657–670 (2014). https://doi.org/10.1007/s00464-013-3226-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-013-3226-7

Keywords

  • Minimally invasive surgery
  • Objective assessment
  • Training
  • Supervised classification
  • TrEndo