Skip to main content

Advertisement

SpringerLink
Log in

Evaluating competency in video-assisted thoracoscopic surgery (VATS) lobectomy performance using a novel assessment tool and virtual reality simulation

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Competency-based training has gained ground in surgical training and with it assessment tools to ensure that training objectives are met. Very few assessment tools are available for evaluating performance in thoracoscopic procedures. Video recordings would provide the possibility of blinded assessment and limited rater bias. This study aimed to provide validity evidence for a newly developed and dedicated tool for assessing competency in Video-Assisted Thoracoscopic Surgery (VATS) lobectomy.

Methods

Participants with varying experience with VATS lobectomy were included from different countries. Video recordings from participants’ performance of a VATS right upper lobe lobectomy on a virtual reality simulator were rated by three raters using a modified version of a newly developed VATS lobectomy assessment tool (the VATSAT) and analyzed in relation to the unitary framework (content, response process, internal structure, relation to other variables, and consequences of testing).

Results

Fifty-three participants performed two consecutive simulated VATS lobectomies on the virtual reality simulator, leaving a total of 106 videos. Content established in previously published studies. Response process Standardized data collection was ensured by using an instructional element, uniform data collection, a special rating program, and automatic generation of the results to a database. Raters were carefully instructed in using the VATSAT, and tryout ratings were carried out. Internal structure Inter-rater reliability was calculated as intra-class correlation coefficients, to 0.91 for average measures (p < 0.001). Test/re-test reliability was calculated as Pearson’s r of 0.70 (p < 0.001). G-coefficient was calculated to be 0.79 with two procedures and three raters. By performing D-theory was found that either three procedures rated by two raters or five procedures rated by one rater were enough to reach an acceptable G-coefficient of ≥ 0.8. Relation to other variables Significant differences between groups were found (p < 0.001). The participants’ VATS lobectomy experience correlated significantly to their VATSAT score (p = 0.016). Consequences of testing The pass/fail score was found to be 14.9 points by the contrasting groups’ method, leaving five false positive (29%) and six false negatives (43%).

Conclusion

Validity evidence was provided for the VATSAT according to the unitary framework. The VATSAT provides supervisors and assessors with a procedure-specific assessment tool for evaluating VATS lobectomy performance and aids with the decision of when the trainee is ready for unsupervised performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Epstein RM, Cassel CK, Epstein RM, de Galan BE, van Gurp PJ, Stuyt PM (2007) Assessment in medical education. N Engl J Med 356:387–396. https://doi.org/10.1056/NEJMra054784

    Article  CAS  PubMed  Google Scholar 

  2. Lodge D, Grantcharov T (2011) Training and assessment of technical skills and competency in cardiac surgery. Eur J Cardio-Thorac Surg 39:287–293. https://doi.org/10.1016/j.ejcts.2010.06.035

    Article  Google Scholar 

  3. Darzi A, Datta V, Mackay S (2001) The challenge of objective assessment of surgical skill. Am J Surg 181:484–486. https://doi.org/10.1016/S0002-9610(01)00624-9

    Article  CAS  PubMed  Google Scholar 

  4. Mcgaghie WC (2015) Mastery learning: it is time for medical education to join the 21st century. Acad Med 90:1–4. https://doi.org/10.1097/ACM.0000000000000911

    Article  Google Scholar 

  5. Sharma B, Mishra A, Aggarwal R, Grantcharov TP (2011) Non-technical skills assessment in surgery. Surg Oncol 20:169–177. https://doi.org/10.1016/j.suronc.2010.10.001

    Article  PubMed  Google Scholar 

  6. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202:469–480. https://doi.org/10.1016/j.amjsurg.2010.10.020

    Article  PubMed  Google Scholar 

  7. Konge L, Lehnert P, Hansen HJ, Petersen RH, Ringsted C (2012) Reliable and valid assessment of performance in thoracoscopy. Surg Endosc 26:1624–1628. https://doi.org/10.1007/s00464-011-2081-7

    Article  PubMed  Google Scholar 

  8. Tong BC, Gustafson MR, Balderson SS, D’Amico TA, Meyerson SL (2012) Validation of a thoracoscopic lobectomy simulator. Eur J Cardio-Thorac Surg 42:364–369. https://doi.org/10.1093/ejcts/ezs012

    Article  Google Scholar 

  9. Konge L, Larsen KR, Clementsen P, Arendrup H, Von Buchwald C, Ringsted C (2012) Reliable and valid assessment of clinical bronchoscopy performance. Respiration 83:53–60. https://doi.org/10.1159/000330061

    Article  PubMed  Google Scholar 

  10. Weizman NF, Manoucheri E, Vitonis AF, Hicks GJ, Einarsson JI, Cohen SL (2015) Design and validation of a novel assessment tool for laparoscopic suturing of the vaginal cuff during hysterectomy. J Surg Educ 72:212–219. https://doi.org/10.1016/j.jsurg.2014.08.015

    Article  PubMed  Google Scholar 

  11. Carlsen CG, Lindorff-Larsen K, Funch-Jensen P, Lund L, Charles P, Konge L (2014) Reliable and valid assessment of Lichtenstein hernia repair skills. Hernia 18:543–548. https://doi.org/10.1007/s10029-013-1196-2

    Article  CAS  PubMed  Google Scholar 

  12. Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A (2008) Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room. Ann Surg 247:372–379. https://doi.org/10.1097/SLA.0b013e318160b371

    Article  PubMed  Google Scholar 

  13. Yang SC, Merrill W (2014) Educational milestone development in phase II specialties: thoracic surgery. j Gr Med Educ 6:329–331

    Article  Google Scholar 

  14. Messick S (1989) Meaning and values in test validation: the science and ethics of assessment. Educ Res 18:5–11. https://doi.org/10.3102/0013189x018002005

    Article  Google Scholar 

  15. Reznick RK, MacRae H (2006) Teaching surgical skills–changes in the wind. N Engl J Med 355:2664–2669. https://doi.org/10.1016/S0084-392X(08)70199-0

    Article  CAS  PubMed  Google Scholar 

  16. Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278. https://doi.org/10.1002/bjs.1800840237

    Article  CAS  PubMed  Google Scholar 

  17. Vogt VY, Givens VM, Keathley CA, Lipscomb GH, Summitt RL (2003) Is a resident’s score on a videotaped objective structured assessment of technical skills affected by revealing the resident’s identity? Am J Obstet Gynecol 189(3):688–691

    Article  PubMed  Google Scholar 

  18. Vassiliou MC, Feldman LS, Fraser S, Charlebois P, Chaudhury P, Stanbridge DD, Fried GM (2007) Evaluating intraoperative laparoscopic skill: direct observation versus blinded videotaped performances. Surg Innov 14:211–216. https://doi.org/10.1177/1553350607308466

    Article  PubMed  Google Scholar 

  19. Jensen K, Bjerrum F, Hansen HJ, Petersen RH, Pedersen JH, Konge L (2016) Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy. Surg Endosc. https://doi.org/10.1007/s00464-016-5254-6

    Article  PubMed  Google Scholar 

  20. Jensen K, Bjerrum F, Hansen HJ, Petersen RH, Pedersen JH, Konge L (2015) A new possibility in thoracoscopic virtual reality simulation training: development and testing of a novel virtual reality simulator for video-assisted thoracoscopic surgery lobectomy. Interact Cardiovasc Thorac Surg 21:420–426. https://doi.org/10.1093/icvts/ivv183

    Article  PubMed  Google Scholar 

  21. Konge L, Ringsted C, Bjerrum F, Tolsgaard MG, Bitsch M, Sørensen JL, Schroeder TV (2015) The simulation centre at Rigshospitalet, Copenhagen, Denmark. J Surg Educ 72:362–365. https://doi.org/10.1016/j.jsurg.2014.11.012

    Article  Google Scholar 

  22. Jensen K, Petersen RH, Hansen HJ, Walker W, Pedersen JH, Konge L (2018) A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy. Surg Endosc. https://doi.org/10.1007/s00464-018-6162-8

    Article  PubMed  Google Scholar 

  23. Subhi Y, Todsen T, Konge L (2014) An integrable, web-based solution for easy assessment of video-recorded performances. Adv Med Educ Pract 5:103–105. https://doi.org/10.2147/AMEP.S62277

    Article  PubMed  PubMed Central  Google Scholar 

  24. Downing SM (2004) the metric of medical education reliability: on the reproducibility of assessment data. Med Educ 38:1006–1012. https://doi.org/10.1046/j.1365-2929.2004.01932.x

    Article  PubMed  Google Scholar 

  25. Petersen RH, Gjeraa K, Jensen K, Møller LB, Hansen HJ, Konge L (2018) Assessment of competence in Video Assisted Thoracoscopic Surgery (VATS) Lobectomy: a Danish nationwide study. J Thorac Cardiovasc Surg. https://doi.org/10.1016/j.jtcvs.2018.04.046

    Article  PubMed  Google Scholar 

  26. Crossley J, Davies H, Humphris G, Jolly B (2002) Generalisability: a key to unlock professional assessment. Med Educ 36:972–978. https://doi.org/10.1046/j.1365-2923.2002.01320.x

    Article  PubMed  Google Scholar 

  27. Konge L, Vilmann P, Clementsen P, Annema JT, Ringsted C (2012) Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer. Endoscopy 44:928–933. https://doi.org/10.1055/s-0032-1309892

    Article  CAS  PubMed  Google Scholar 

  28. Mahmood O, Dagnæs J, Bube S, Rohrsted M, Konge L (2017) Nonspecialist raters can provide reliable assessments of procedural skills. J Surg Educ. https://doi.org/10.1016/j.jsurg.2017.07.003

    Article  PubMed  Google Scholar 

  29. Våpenstad C, Buzink SN (2013) Procedural virtual reality simulation in minimally invasive surgery. Surg Endosc 27:364–377. https://doi.org/10.1007/s00464-012-2503-1

    Article  PubMed  Google Scholar 

  30. Walsh CM, Ling SC, Khanna N, Cooper MA, Grover SC, May G, Walters TD, Rabeneck L, Reznick R, Carnahan H (2014) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 79:798–807.e5. https://doi.org/10.1016/j.gie.2013.10.035

    Article  PubMed  Google Scholar 

  31. Gjeraa K, Spanager L, Konge L, Petersen RH, Ostergaard D (2016) Non-technical skills in minimally invasive surgery teams: a systematic review. Surg Endosc. https://doi.org/10.1007/s00464-016-4890-1

    Article  PubMed  Google Scholar 

  32. Hashimoto DA, Phitayakorn R, Fernandez-del Castillo C, Meireles O (2016) A blinded assessment of video quality in wearable technology for telementoring in open surgery: the Google Glass experience. Surg Endosc 30:372–378. https://doi.org/10.1007/s00464-015-4178-x

    Article  PubMed  Google Scholar 

  33. Vallurupalli S, Paydak H, Agarwal SK, Agrawal M, Assad-Kottner C (2013) Wearable technology to improve education and patient outcomes in a cardiology fellowship program—a feasibility study. Health Technol 3:267–270. https://doi.org/10.1007/s12553-013-0065-4

    Article  Google Scholar 

  34. Hopmans CJ, den Hoed PT, van der Laan L, van der Harst E, van der Elst M, Mannaerts GHH, Dawson I, Timman R, Wijnhoven BPL, IJzermans JNM (2014) Assessment of surgery residents’ operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study. Surgery 156:1078–1088. https://doi.org/10.1016/j.surg.2014.04.052

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Cardiothoracic Surgery, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark

    Katrine Jensen, Henrik Jessen Hansen, René Horsleben Petersen & Jesper Holst Pedersen

  2. Copenhagen Academy for Medical Education and Simulation (CAMES), University of Copenhagen and the Capital Region, Copenhagen, Denmark

    Katrine Jensen & Lars Konge

  3. Department of Cardiothoracic and Vascular Surgery, Odense University Hospital, Odense, Denmark

    Kirsten Neckelmann

  4. Department of Cardiothoracic and Vascular Surgery, Aarhus University Hospital, Skejby, Denmark

    Henrik Vad

  5. Department of Cardiothoracic Surgery, Aalborg University Hospital, Aalborg, Denmark

    Lars Borgbjerg Møller

  6. Department 5404, Copenhagen Academy for Medical Education and Simulation (CAMES), Blegdamsvej 9, 2100, Copenhagen, Denmark

    Katrine Jensen

Authors
  1. Katrine Jensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henrik Jessen Hansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. René Horsleben Petersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kirsten Neckelmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Henrik Vad
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lars Borgbjerg Møller
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jesper Holst Pedersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lars Konge
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katrine Jensen.

Ethics declarations

Disclosures

Katrine Jensen’s salary during her Ph.D. was partly funded by The Danish Cancer Society (Kræftens Bekæmpelse, “Knæk Cancer”), and this study was carried out as part of her Ph.D. Henrik Jessen Hansen and René Horsleben Petersen are at the Speakers Bureau of Medtronic. Kirsten Neckelmann, Henrik Vad, Lars Møller, Jesper Holst Pedersen, and Lars Konge have no conflicts of interest or financial ties to disclose.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jensen, K., Hansen, H.J., Petersen, R.H. et al. Evaluating competency in video-assisted thoracoscopic surgery (VATS) lobectomy performance using a novel assessment tool and virtual reality simulation. Surg Endosc 33, 1465–1473 (2019). https://doi.org/10.1007/s00464-018-6428-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-018-6428-1

Keywords

Search

Navigation