Skip to main content
SpringerLink
Log in

Subjective vs. objective assessment of simulation performance on laparoscopic cholecystectomy: are we evaluating the right things?

  • 2021 SAGES Oral
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Simulation using virtual reality (VR) simulators is an important tool in surgical training. VR laparoscopic simulators can provide immediate objective performance assessment without observer evaluation. This study aims to explore the correlation between subjective observer evaluation and VR laparoscopic simulator performance metrics in a laparoscopic cholecystectomy (LC) simulation module.

Methods

A LC simulation module using a VR laparoscopic simulator was completed by PGY2–3 general surgery residents at a single institution. Simulation performance was recorded and evaluated by a trained evaluator using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) form, the Objective Structured Assessment of Technical Skills (OSATS) form, and a LC-specific simulation assessment form (LC-SIM). Objective performance metrics were also obtained from the simulator system. Performance before the curriculum (pre-test) and after the curriculum (post-test) were compared.

Results

Fourteen residents were included in the study. There were significant improvements from pre-test to post-test on each component of GOALS, OSATS, and LC-SIM scores (all p values < 0.05). In terms of objective simulator metrics, significant improvements were noted in time to extract gallbladder (481 ± 221 vs 909 ± 366 min, p = 0.019), total number of movements (475 ± 264 vs 839 ± 324 min, p = 0.012), and total path length (955 ± 475 vs 1775 ± 632 cm, p = 0.012) from pre-test to post-test. While number of movements and total path lengths of both hands decreased, speed of right instrument also decreased from 4.1 + 2.7 to 3.0 ± 0.7 cm/sec (p = 0.007). Average speed of left instrument was associated with respect for tissue (r = 0.60, p < 0.05) and depth perception (r = 0.68, p < 0.05) on post-test evaluations.

Conclusion

Our study demonstrated significant improvement in technical skills based on subjective evaluator assessment as well as objective simulator metrics after simulation. The few correlations identified between the subjective evaluator and the objective simulator assessments suggest the two evaluation modalities were measuring different aspects of the technical skills and should both be considered in the evaluation process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou M, Davidson BR. 2013 Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev Doi: https://doi.org/10.1002/14651858.CD006575.pub3

  2. Walsh CM, Sherlock ME, Ling SC, Carnahan H. 2012 Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst Rev 13;(6): e008237. doi: https://doi.org/10.1002/14651858.CD008237.pub2. Update in: Cochrane Database Syst Rev. 2018 Aug 17;8:CD008237. PMID: 22696375

  3. Abdelsattar JM, AlJamal YN, Ruparel RK, Rowse PG, Heller SF, Farley DR (2018) Correlation of objective assessment data with general surgery resident in-training evaluation reports and operative volumes. J Surg Educ 75:1430–1436

    Article  Google Scholar 

  4. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A (2009) Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg 96(9):1086–1093. https://doi.org/10.1002/bjs.6679

    Article  CAS  PubMed  Google Scholar 

  5. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278. https://doi.org/10.1046/j.1365-2168.1997.02502.x

    Article  CAS  PubMed  Google Scholar 

  6. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, Fried GM (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107–113

    Article  Google Scholar 

  7. Williams RG, Sanfey H, Chen XP, Dunnington GL (2012) A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment: a controlled prospective observational study. Ann Surg 256(1):177–187. https://doi.org/10.1097/SLA.0b013e31825b6de4 (PMID: 22751518)

    Article  PubMed  Google Scholar 

  8. Aggarwal R, Grantcharov TP, Eriksen JR, Blirup D, Kristiansen VB, Funch-Jensen P, Darzi A (2006) An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 244(2):310–314. https://doi.org/10.1097/01.sla.0000218094.92650.44.PMID:16858196;PMCID:PMC1602164

    Article  PubMed  PubMed Central  Google Scholar 

  9. Wilson M, McGrath J, Vine S, Brewer J, Defriend D, Masters R. 2010 Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts. Surg Endosc 24(10): 2458–64 doi: https://doi.org/10.1007/s00464-010-0986-1

  10. Bell AK, Zhou M, Schwaitzberg SD, Cao CG (2009) Using a dynamic training environment to acquire laparoscopic surgery skill. Surg Endosc 23(10):2356–2363. https://doi.org/10.1007/s00464-009-0346-1

    Article  CAS  PubMed  Google Scholar 

  11. Binkley J, Bukoski AD, Doty J, Crane M, Barnes SL, Quick JA (2019) Surgical simulation: markers of proficiency. J Surg Educ 76:234–241

    Article  Google Scholar 

Download references

Acknowledgements

This manuscript was accepted for an oral presentation at the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) 2021 Annual Meeting in Las Vegas, NV from August 31st to September 3rd, 2021

Funding

None.

Author information

Authors and Affiliations

  1. Department of Surgery, NorthShore University HealthSystem, Evanston, IL, USA

    Yohei Kojima, Harry J. Wong, Kristine Kuchta, John G. Linn, Stephen P. Haggerty, Woody Denham & Michael B. Ujiki

  2. Department of Surgery, University of Chicago Medicine, Chicago, IL, USA

    Harry J. Wong

  3. Department of Surgery, Kyorin University, Tokyo, Japan

    Yohei Kojima

Authors
  1. Yohei Kojima
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harry J. Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kristine Kuchta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John G. Linn
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephen P. Haggerty
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Woody Denham
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael B. Ujiki
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Harry J. Wong.

Ethics declarations

Disclosures

Drs Kojima, Wong, and Kristine Kuchta have no conflicts of interest or financial ties to disclose. Dr Ujiki receives grant funding from Medtronic [ERP-2020 1228]. Drs Linn, Haggerty, and Ujiki receive payment for lectures from Gore. Dr Haggerty is a paid consultant to Medtronic. Dr Ujiki is a board member for Boston Scientific, is a paid consultant for Olympus and Cook, and receives payment for lectures from Medtronic and Erbe.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kojima, Y., Wong, H.J., Kuchta, K. et al. Subjective vs. objective assessment of simulation performance on laparoscopic cholecystectomy: are we evaluating the right things?. Surg Endosc 36, 6661–6671 (2022). https://doi.org/10.1007/s00464-021-08936-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-021-08936-7

Keywords

Search

Navigation