Skip to main content

Laparoscopic Surgical Skills Assessment: Can Simulators Replace Experts?

World Journal of Surgery volume 33, pages 440–447 (2009)Cite this article

Abstract

Introduction

Global Rating Scales (GRS) quantify and structure subjective expert assessment of skill. Hybrid simulators measure performance during physical laparoscopic tasks through instrument motion analysis. We assessed whether motion analysis metrics were as accurate as structured expert opinion by using GRS.

Methods

A random sample of 10 consultant laparoscopic surgeons, 10 senior trainees, and 10 novice students were assessed on a Sharp Dissection task. Coded video footage was reviewed by two blinded assessors and scored using a Likert Scale. Correlation with metrics was tested using Spearman’s rho. Inter-rater reliability was measured using intraclass correlation coefficient (ICC).

Results

Strongest GRS–Metric correlations were found for Time/Motion/Progress with Time (Spearman’s rho 0.88; p < 0.05) and Instrument Handling with Path Length (Spearman’s rho 0.8; p < 0.05). Smoothness correlated with Respect for Tissue in Rater 1 (rho 0.68) but not Rater 2 (rho 0.18). Mean GRS showed stronger inter-rater agreement than individual scale components (ICC 0.68). Correlation coefficients with actual experience group were 0.58–0.74 for mean GRS score and 0.67–0.78 for metrics (Spearman’s rho, p < 0.05).

Conclusions

Metrics correlate well with GRS assessment, supporting concurrent validity. Metrics predict experience level as accurately as global rating and are construct valid. Hybrid simulators could provide resource-efficient feedback, freeing trainers to concentrate on teaching.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Darzi A, Sminth S, Taffinder N (1999) Assessing operative skill: needs to become more objective. Br Med J 318:888–889

    Google Scholar 

  2. Darzi A, Mackay S (2001) Assessment of surgical competence. Qual Health Care 10:ii64–ii69

    PubMed  Article  Google Scholar 

  3. Figert P, Park A, Witkze D, Schwartz R (2001) Transfer of training in acquiring laparoscopic skills. J Am Coll Surg 193:534–537

    Article  Google Scholar 

  4. MacFadyen BV Jr, Vecchio R, Ricardo AE, Mathis CR (1998) Bile duct injury after laparoscopic cholecystectomy: the United States experience. Surg Endosc 12:315–321

    PubMed  Article  Google Scholar 

  5. Aggarwal R, Grantcharov TP, Moorthy K, Hance JR, Darzi A (2006) A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. Am J Surg 191:128–133

    PubMed  Article  Google Scholar 

  6. Broe D, Ridgway P, Johnson S, Tierney S, Conlon C (2006) Construct validation of a novel hybrid surgical simulator. Surg Endosc 20:900–904

    PubMed  Article  CAS  Google Scholar 

  7. Champion HR, Gallagher AG (2003) Surgical simulation: a “good idea whose time has come”. Br J Surg 90:767–768

    PubMed  Article  CAS  Google Scholar 

  8. Duffy A, Hogle N, McCarthy H, Lew J, Egan A, Christos P, Fowler D (2005) Construct Validity for the LAPSIM laparoscopic surgical simulator. Surg Endosc 19:401–405

    PubMed  Article  CAS  Google Scholar 

  9. Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen P (2001) Virtual reality computer simulation: an objective method for the evaluation of laparoscopic surgical skills. Surg Endosc 15:242–244

    PubMed  Article  CAS  Google Scholar 

  10. Carter F, Schijven MP, Aggarwal R, Grantcharov TP, Francis N, Hanna G, Jakimowicz J (2005) Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc 19:1523–1532

    PubMed  Article  CAS  Google Scholar 

  11. Martin J, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278

    PubMed  Article  CAS  Google Scholar 

  12. Reznick R, Regehr G, MacRae H, Martin J, McCulloch RN (1997) Testing technical skill via an innovative “bench station” examination. Am J Surg 173:226–230

    PubMed  Article  CAS  Google Scholar 

  13. Colton T (1974) Statistics in medicine. Little, Brown and Company, Boston

    Google Scholar 

  14. Szabo Z, Hunter J, Berci G, Sackier J, Cushieri A (1994) Analysis of surgical movements during suturing in laparoscopy. Endosc Surg Allied Technol 2:55–61

    PubMed  CAS  Google Scholar 

  15. Satava RM, Gallagher AG, Pelligrini CA (2003) Surgical competence and surgical proficiency: definitions, taxonomy and metrics. J Am Coll Surg 196:933–937

    PubMed  Article  Google Scholar 

  16. Satava RM, Cushieri A, Hamdorf J (2005) Metrics for objective assessment: preliminary summary of the Surgical Skills Workshop. Surg Endosc 17:220–226

    Article  Google Scholar 

  17. Chaudhry A, Sutton C, Wood J, Stone R, McCloy R (1999) Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human-computer interface. Ann Royal Coll Surg 81:281–286

    CAS  Google Scholar 

  18. Gallagher AG, Ritchie K, McClure N, McGuigan J (2001) Objective psychomotor assessment of senior, junior and novice laparoscopists with virtual reality. World J Surg 25:1478–1483

    PubMed  Article  CAS  Google Scholar 

  19. Gallagher AG, Smith CD, Bowers SP, Seymour NE, Pearson A, McNatt S, Hananel D, Satava RM (2003) Psychomotor skills assessment in practicing surgeons experienced in performing advanced laparoscopic procedures. J Am Coll Surg 197:479–488

    PubMed  Article  Google Scholar 

  20. Schijven MP, Jakimowicz J, Schot C (2002) The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) objectifying subjective psychomotor test performance. Surg Endosc 16:943–948

    PubMed  Article  CAS  Google Scholar 

  21. Datta V, Chang A, Mackay S, Darzi A (2002) The relationship between motion analysis and surgical technical assessments. Am J Surg 184:70–73

    PubMed  Article  Google Scholar 

  22. Moorthy K, Munz Y, Dosis A, Bello F, Chang A, Darzi A (2004) Bimodal assessment of laparoscopic suturing skills: construct and concurrent validity. Surg Endosc 18:1608–1612

    PubMed  CAS  Google Scholar 

  23. Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A (2004) Laparoscopic virtual reality and box trainers. Surg Endosc 18:485–494

    PubMed  Article  CAS  Google Scholar 

  24. Vassiliou M, Feldman L, Andrew C, Bergman S, Leffondre K, Stanbridge D, Fried GM (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107–113

    PubMed  Article  Google Scholar 

  25. Bann S, Mackay Davis I, Moorthy K, Munz Y, Hernandez J, Khan M, Datta V, Darzi A (2005) The reliability of multiple measures of surgery and the role of human performance. Am J Surg 189:747–752

    PubMed  Article  Google Scholar 

  26. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91:146–150

    PubMed  Article  CAS  Google Scholar 

  27. Pellen MGC, Horgan LF, Barton JR, Attwood SEA (2008) Construct validity of the ProMIS laparoscopic simulator. Surg Endosc [Epub ahead of print]

  28. Dath D, Regehr G, Birch D, Schlachta C, Poulin E, Mamazza J, Reznick R, MacRae H (2004) Toward reliable operative assessment. Surg Endosc 18:1800–1804

    PubMed  Article  CAS  Google Scholar 

  29. Scott DJ, Rege RV, Bergen PC, Guo WA, Laycock R, Tesfay ST, Valentine RJ, Jones DB (2000) Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech 10:183–190

    Article  CAS  Google Scholar 

  30. Rosser JC, Rosser LE, Savalgi RS (1997) Skill acquisition and assessment for laparoscopic surgery. Arch Surg 132:200–204

    PubMed  CAS  Google Scholar 

  31. Scott D, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, Euhus D, Jeyarajah R, Thompson W, Jones DB (2000) Laparoscopic training on bench models: better and more cost effective than operating room experience. J Am Coll Surg 191:272–283

    PubMed  Article  CAS  Google Scholar 

  32. Vassiliou M, Ghitulescu G, Feldman L, Stanbridge D, Leffondre K, Sigman HH, Fried GM (2006) The MISTELS program to measure technical skill in laparoscopic surgery. Surg Endosc 20:744–747

    PubMed  Article  CAS  Google Scholar 

  33. Fried GM, Feldman L, Vassiliou M, Fraser S, Stanbridge D, Ghitulescu G, Andrew C (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–528

    PubMed  Article  Google Scholar 

  34. Fichera A, Prachand V, Kives S, Levine R, Hasson H (2005) Physical reality simulation for training of laparoscopists in the 21st century. a multispecialty multi-institutional study. J Soc Laparoscopic Surg 9:125–129

    Google Scholar 

  35. Derossis AM, Bothwell J, Sigman HH, Fried GM (1998) The effect of practice on performance in a laparoscopic simulator. Surg Endosc 12:1117–1120

    PubMed  Article  CAS  Google Scholar 

  36. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Anderson DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236:458–464

    PubMed  Article  Google Scholar 

Download references

Author information

Affiliations

  1. Department of General Surgery, Northumbria Healthcare NHS Trust, Rake Lane, North Shields, UK

    Michael Pellen, Liam Horgan & Stephen Attwood

  2. Department of Gastroenterology, Northumbria Healthcare NHS Trust, Rake Lane, North Shields, UK

    J. Roger Barton

Authors
  1. Michael Pellen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liam Horgan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Roger Barton
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephen Attwood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Pellen.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Pellen, M., Horgan, L., Roger Barton, J. et al. Laparoscopic Surgical Skills Assessment: Can Simulators Replace Experts?. World J Surg 33, 440–447 (2009). https://doi.org/10.1007/s00268-008-9866-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00268-008-9866-4

Keywords

  • Intraclass Correlation Coefficient
  • Laparoscopic Skill
  • Hybrid Simulator
  • Global Rate Scales
  • Penalty Score