Skip to main content
SpringerLink
Log in

Construct validity of individual and summary performance metrics associated with a computer-based laparoscopic simulator

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Computer-based surgical simulators capture a multitude of metrics based on different aspects of performance, such as speed, accuracy, and movement efficiency. However, without rigorous assessment, it may be unclear whether all, some, or none of these metrics actually reflect technical skill, which can compromise educational efforts on these simulators. We assessed the construct validity of individual performance metrics on the LapVR simulator (Immersion Medical, San Jose, CA, USA) and used these data to create task-specific summary metrics.

Methods

Medical students with no prior laparoscopic experience (novices, N = 12), junior surgical residents with some laparoscopic experience (intermediates, N = 12), and experienced surgeons (experts, N = 11) all completed three repetitions of four LapVR simulator tasks. The tasks included three basic skills (peg transfer, cutting, clipping) and one procedural skill (adhesiolysis).

Results

We selected 36 individual metrics on the four tasks that assessed six different aspects of performance, including speed, motion path length, respect for tissue, accuracy, task-specific errors, and successful task completion. Four of seven individual metrics assessed for peg transfer, six of ten metrics for cutting, four of nine metrics for clipping, and three of ten metrics for adhesiolysis discriminated between experience levels. Time and motion path length were significant on all four tasks. We used the validated individual metrics to create summary equations for each task, which successfully distinguished between the different experience levels.

Conclusion

Educators should maintain some skepticism when reviewing the plethora of metrics captured by computer-based simulators, as some but not all are valid. We showed the construct validity of a limited number of individual metrics and developed summary metrics for the LapVR. The summary metrics provide a succinct way of assessing skill with a single metric for each task, but require further validation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Hawasli A, Featherstone R, Lloyd L, Vorhees M (1996) Laparoscopic training in residency program. J Laparoendosc Surg 6:171–174

    Article  CAS  PubMed  Google Scholar 

  2. Bridges M, Diamond DL (1999) The financial impact of teaching surgical residents in the operating room. Am J Surg 177:28–32

    Article  CAS  PubMed  Google Scholar 

  3. Philibert I, Friedmann P, Williams WT (2002) New requirements for resident duty hours. JAMA 288:1112–1114

    Article  PubMed  Google Scholar 

  4. Beyer L, Troyer JD, Mancini J, Bladou F, Berdah SV, Karsenty G (2011) Impact of laparoscopy simulator training on the technical skills of future surgeons in the operating room: a prospective study. Am J Surg 202:265–272

    Article  PubMed  Google Scholar 

  5. Fried GM, Feldman LS, Vassiliou MC, Fraser SA, Stanbridge D, Ghitulescu G, Andrew CG (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–528

    Article  PubMed Central  PubMed  Google Scholar 

  6. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91:146–150

    Article  CAS  PubMed  Google Scholar 

  7. Park J, MacRae H, Musselman LJ, Rossos P, Hamstra SJ, Wolman S, Reznick RK (2007) Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J Surg 194:205–211

    Article  PubMed  Google Scholar 

  8. Scott DJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, Euhus DM, Jeyarajah DR, Thompson WM, Jones DB (2000) Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg 191:272–283

    Article  CAS  PubMed  Google Scholar 

  9. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236:458–463

    Article  PubMed Central  PubMed  Google Scholar 

  10. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez R, Fried GM (2010) Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg 199:115–120

    Article  PubMed  Google Scholar 

  11. Derossis AM, Bothwell J, Sigman HH, Fried GM (1998) The effect of practice on performance in a laparoscopic simulator. Surg Endosc 12:1117–1120

    Article  CAS  PubMed  Google Scholar 

  12. Calatayud D, Arora S, Aggarwal R, Kruglikova I, Schulze S, Funch-Jensen P, Grantcharov T (2010) Warm-up in a virtual reality environment improves performance in the operating room. Ann Surg 251:1181–1185

    Article  PubMed  Google Scholar 

  13. Diesen DL, Erhunmwunsee L, Bennett KM, Ben-David K, Yurcisin B, Ceppa EP, Omotosho PA, Perez A, Pryor A (2011) Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic surgery training of novices. J Surg Educ 68:282–289

    Article  PubMed  Google Scholar 

  14. Gaba DM (2007) The future vision of simulation in healthcare. Simul Healthc 2:126–135

    Article  PubMed  Google Scholar 

  15. Satava RM, Gallagher AG, Pellegrini CA (2003) Surgical competence and surgical proficiency: definitions, taxonomy, and metrics. J Am Coll Surg 196:933–937

    Article  PubMed  Google Scholar 

  16. Downing SM (2003) Validity: on meaningful interpretation of assessment data. Med Educ 37:830–837

    Article  PubMed  Google Scholar 

  17. Carter FJ, Schijven MP, Aggarwal R, Grantcharov T, Francis NK, Hanna GB, Jakimowicz JJ (2005) Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc 19:1523–1532

    Article  CAS  PubMed  Google Scholar 

  18. Edelman DA, Mattos MA, Bouwman DL (2010) FLS skill retention (learning) in first year surgery residents. J Surg Res 163:24–28

    Article  PubMed  Google Scholar 

  19. Sherman V, Feldman LS, Stanbridge D, Kazmi R, Fried GM (2005) Assessing the learning curve for the acquisition of laparoscopic skills on a virtual reality simulator. Surg Endosc 19:678–682

    Article  CAS  PubMed  Google Scholar 

  20. Satava RM, Cuschieri A, Hamdorf J (2003) Metrics for objective assessment. Surg Endosc 17:220–226

    Article  CAS  PubMed  Google Scholar 

  21. Grober ED, Hamstra SJ, Wanzel KR, Reznick RK, Matsumoto ED, Sidhu RS, Jarvi KA (2004) The educational impact of bench model fidelity on the acquisition of technical skill: the use of clinically relevant outcome measures. Ann Surg 240:374–381

    Article  PubMed Central  PubMed  Google Scholar 

  22. Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD (2002) The effect of bench model fidelity on endourological skills: a randomized controlled study. J Urol 167:1243–1247

    Article  PubMed  Google Scholar 

  23. Stefanidis D, Korndorffer JR Jr, Sierra R, Touchard C, Dunne JB, Scott DJ (2005) Skill retention following proficiency-based laparoscopic simulator training. Surgery 138:165–170

    Article  PubMed  Google Scholar 

  24. Woodrum DT, Andreatta PB, Yellamanchilli RK, Feryus L, Gauger PG, Minter RM (2006) Construct validity of the LapSim laparoscopic surgical simulator. Am J Surg 191:28–32

    Article  PubMed  Google Scholar 

  25. Eriksen JR, Grantcharov T (2005) Objective assessment of laparoscopic skills using a virtual reality stimulator. Surg Endosc 19:1216–1219

    Article  CAS  PubMed  Google Scholar 

  26. Bell RH (2007) Surgical council on resident education: a new organization devoted to graduate surgical education. J Am Coll Surg 204:341–346

    Article  PubMed  Google Scholar 

  27. Hamilton EC, Scott DJ, Fleming JB, Rege RV, Laycock R, Bergen PC, Tesfay ST, Jones DB (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc 16:406–411

    Article  CAS  PubMed  Google Scholar 

  28. van Dongen KW, Tournoij E, van der Zee DC, Schijven MP, Broeders IA (2007) Construct validity of the LapSim: can the LapSim virtual reality simulator distinguish between novices and experts? Surg Endosc 21:1413–1417

    Article  PubMed  Google Scholar 

  29. Scott DJ, Ritter EM, Tesfay ST, Pimentel EA, Nagji A, Fried GM (2008) Certification pass rate of 100% for fundamentals of laparoscopic surgery skills after proficiency-based training. Surg Endosc 22:1887–1893

    Article  PubMed  Google Scholar 

  30. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM (2005) Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 241:364–372

    Article  PubMed Central  PubMed  Google Scholar 

  31. American Educational Research Association, American Psychological Association, National Council on Measurement in Education (1999) Standards for educational and psychological testing. American Psychological Association, Washington, DC

    Google Scholar 

  32. Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J (2003) Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. Am J Surg 185:146–149

    Article  PubMed  Google Scholar 

  33. McDougall EM, Corica FA, Boker JR, Sala LG, Stoliar G, Borin JF, Chu FT, Clayman RV (2006) Construct validity testing of a laparoscopic surgical simulator. J Am Coll Surg 202:779–787

    Article  PubMed  Google Scholar 

  34. Iwata N, Fujiwara M, Kodera Y, Tanaka C, Ohashi N, Nakayama G, Koike M, Nakao A (2011) Construct validity of the LapVR virtual-reality surgical simulator. Surg Endosc 25:423–428

    Article  PubMed  Google Scholar 

  35. Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL (1998) Development of a model for training and evaluation of laparoscopic skills. Am J Surg 175:482–487

    Article  CAS  PubMed  Google Scholar 

  36. Stefanidis D, Scott DJ, Korndorffer JR Jr (2009) Do metrics matter? Time versus motion tracking for performance assessment of proficiency-based laparoscopic skills training. Simul Healthc 4:104–108

    Article  PubMed  Google Scholar 

  37. Gauger PG, Hauge LS, Andreatta PB, Hamstra SJ, Hillard ML, Arble EP, Kasten SJ, Mullan PB, Cederna PS, Minter RM (2010) Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg 199:72–80

    Article  PubMed  Google Scholar 

  38. Aggarwal R, Ward J, Balasundaram I, Sains P, Athanasiou T, Darzi A (2007) Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg 246:771–779

    Article  PubMed  Google Scholar 

  39. Gallagher AG, Lederman AB, McGlade K, Satava RM, Smith CD (2004) Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc 18:660–665

    Article  CAS  PubMed  Google Scholar 

Download references

Disclosure

Justin Rivard, Ashley Vergis, Bertram Unger, Krista Hardy, Chris Andrew, Lawrence Gillman, and Jason Park have no conflicts of interest or financial ties to disclose. Funding was received from the Canadian Association of General Surgeons Research Fund by Ethicon Endosurgery.

Author information

Authors and Affiliations

  1. Department of Surgery, St. Boniface General Hospital, University of Manitoba, 3rd Floor Z-Block, Winnipeg, MB, R2H 2A6, Canada

    Justin D. Rivard, Ashley S. Vergis, Krista M. Hardy, Chris G. Andrew, Lawrence M. Gillman & Jason Park

  2. Department of Medical Education, University of Manitoba, Winnipeg, MB, Canada

    Bertram J. Unger

Authors
  1. Justin D. Rivard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ashley S. Vergis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bertram J. Unger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Krista M. Hardy
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chris G. Andrew
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lawrence M. Gillman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jason Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Justin D. Rivard.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rivard, J.D., Vergis, A.S., Unger, B.J. et al. Construct validity of individual and summary performance metrics associated with a computer-based laparoscopic simulator. Surg Endosc 28, 1921–1928 (2014). https://doi.org/10.1007/s00464-013-3414-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-013-3414-5

Keywords

Search

Navigation