Abstract
Introduction
Global Rating Scales (GRS) quantify and structure subjective expert assessment of skill. Hybrid simulators measure performance during physical laparoscopic tasks through instrument motion analysis. We assessed whether motion analysis metrics were as accurate as structured expert opinion by using GRS.
Methods
A random sample of 10 consultant laparoscopic surgeons, 10 senior trainees, and 10 novice students were assessed on a Sharp Dissection task. Coded video footage was reviewed by two blinded assessors and scored using a Likert Scale. Correlation with metrics was tested using Spearman’s rho. Inter-rater reliability was measured using intraclass correlation coefficient (ICC).
Results
Strongest GRS–Metric correlations were found for Time/Motion/Progress with Time (Spearman’s rho 0.88; p < 0.05) and Instrument Handling with Path Length (Spearman’s rho 0.8; p < 0.05). Smoothness correlated with Respect for Tissue in Rater 1 (rho 0.68) but not Rater 2 (rho 0.18). Mean GRS showed stronger inter-rater agreement than individual scale components (ICC 0.68). Correlation coefficients with actual experience group were 0.58–0.74 for mean GRS score and 0.67–0.78 for metrics (Spearman’s rho, p < 0.05).
Conclusions
Metrics correlate well with GRS assessment, supporting concurrent validity. Metrics predict experience level as accurately as global rating and are construct valid. Hybrid simulators could provide resource-efficient feedback, freeing trainers to concentrate on teaching.
Similar content being viewed by others
References
Darzi A, Sminth S, Taffinder N (1999) Assessing operative skill: needs to become more objective. Br Med J 318:888–889
Darzi A, Mackay S (2001) Assessment of surgical competence. Qual Health Care 10:ii64–ii69
Figert P, Park A, Witkze D, Schwartz R (2001) Transfer of training in acquiring laparoscopic skills. J Am Coll Surg 193:534–537
MacFadyen BV Jr, Vecchio R, Ricardo AE, Mathis CR (1998) Bile duct injury after laparoscopic cholecystectomy: the United States experience. Surg Endosc 12:315–321
Aggarwal R, Grantcharov TP, Moorthy K, Hance JR, Darzi A (2006) A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. Am J Surg 191:128–133
Broe D, Ridgway P, Johnson S, Tierney S, Conlon C (2006) Construct validation of a novel hybrid surgical simulator. Surg Endosc 20:900–904
Champion HR, Gallagher AG (2003) Surgical simulation: a “good idea whose time has come”. Br J Surg 90:767–768
Duffy A, Hogle N, McCarthy H, Lew J, Egan A, Christos P, Fowler D (2005) Construct Validity for the LAPSIM laparoscopic surgical simulator. Surg Endosc 19:401–405
Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen P (2001) Virtual reality computer simulation: an objective method for the evaluation of laparoscopic surgical skills. Surg Endosc 15:242–244
Carter F, Schijven MP, Aggarwal R, Grantcharov TP, Francis N, Hanna G, Jakimowicz J (2005) Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc 19:1523–1532
Martin J, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278
Reznick R, Regehr G, MacRae H, Martin J, McCulloch RN (1997) Testing technical skill via an innovative “bench station” examination. Am J Surg 173:226–230
Colton T (1974) Statistics in medicine. Little, Brown and Company, Boston
Szabo Z, Hunter J, Berci G, Sackier J, Cushieri A (1994) Analysis of surgical movements during suturing in laparoscopy. Endosc Surg Allied Technol 2:55–61
Satava RM, Gallagher AG, Pelligrini CA (2003) Surgical competence and surgical proficiency: definitions, taxonomy and metrics. J Am Coll Surg 196:933–937
Satava RM, Cushieri A, Hamdorf J (2005) Metrics for objective assessment: preliminary summary of the Surgical Skills Workshop. Surg Endosc 17:220–226
Chaudhry A, Sutton C, Wood J, Stone R, McCloy R (1999) Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human-computer interface. Ann Royal Coll Surg 81:281–286
Gallagher AG, Ritchie K, McClure N, McGuigan J (2001) Objective psychomotor assessment of senior, junior and novice laparoscopists with virtual reality. World J Surg 25:1478–1483
Gallagher AG, Smith CD, Bowers SP, Seymour NE, Pearson A, McNatt S, Hananel D, Satava RM (2003) Psychomotor skills assessment in practicing surgeons experienced in performing advanced laparoscopic procedures. J Am Coll Surg 197:479–488
Schijven MP, Jakimowicz J, Schot C (2002) The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) objectifying subjective psychomotor test performance. Surg Endosc 16:943–948
Datta V, Chang A, Mackay S, Darzi A (2002) The relationship between motion analysis and surgical technical assessments. Am J Surg 184:70–73
Moorthy K, Munz Y, Dosis A, Bello F, Chang A, Darzi A (2004) Bimodal assessment of laparoscopic suturing skills: construct and concurrent validity. Surg Endosc 18:1608–1612
Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A (2004) Laparoscopic virtual reality and box trainers. Surg Endosc 18:485–494
Vassiliou M, Feldman L, Andrew C, Bergman S, Leffondre K, Stanbridge D, Fried GM (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107–113
Bann S, Mackay Davis I, Moorthy K, Munz Y, Hernandez J, Khan M, Datta V, Darzi A (2005) The reliability of multiple measures of surgery and the role of human performance. Am J Surg 189:747–752
Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91:146–150
Pellen MGC, Horgan LF, Barton JR, Attwood SEA (2008) Construct validity of the ProMIS laparoscopic simulator. Surg Endosc [Epub ahead of print]
Dath D, Regehr G, Birch D, Schlachta C, Poulin E, Mamazza J, Reznick R, MacRae H (2004) Toward reliable operative assessment. Surg Endosc 18:1800–1804
Scott DJ, Rege RV, Bergen PC, Guo WA, Laycock R, Tesfay ST, Valentine RJ, Jones DB (2000) Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech 10:183–190
Rosser JC, Rosser LE, Savalgi RS (1997) Skill acquisition and assessment for laparoscopic surgery. Arch Surg 132:200–204
Scott D, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, Euhus D, Jeyarajah R, Thompson W, Jones DB (2000) Laparoscopic training on bench models: better and more cost effective than operating room experience. J Am Coll Surg 191:272–283
Vassiliou M, Ghitulescu G, Feldman L, Stanbridge D, Leffondre K, Sigman HH, Fried GM (2006) The MISTELS program to measure technical skill in laparoscopic surgery. Surg Endosc 20:744–747
Fried GM, Feldman L, Vassiliou M, Fraser S, Stanbridge D, Ghitulescu G, Andrew C (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–528
Fichera A, Prachand V, Kives S, Levine R, Hasson H (2005) Physical reality simulation for training of laparoscopists in the 21st century. a multispecialty multi-institutional study. J Soc Laparoscopic Surg 9:125–129
Derossis AM, Bothwell J, Sigman HH, Fried GM (1998) The effect of practice on performance in a laparoscopic simulator. Surg Endosc 12:1117–1120
Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Anderson DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236:458–464
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pellen, M., Horgan, L., Roger Barton, J. et al. Laparoscopic Surgical Skills Assessment: Can Simulators Replace Experts?. World J Surg 33, 440–447 (2009). https://doi.org/10.1007/s00268-008-9866-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00268-008-9866-4