To assess the construct and face validity of ArthroS, a passive haptic VR simulator. A secondary aim was to evaluate the novel performance metrics produced by this simulator.
Two groups of 30 participants, each divided into novice, intermediate or expert based on arthroscopic experience, completed three separate tasks on either the knee or shoulder module of the simulator. Performance was recorded using 12 automatically generated performance metrics and video footage of the arthroscopic procedures. The videos were blindly assessed using a validated global rating scale (GRS). Participants completed a survey about the simulator’s realism and training utility.
This new simulator demonstrated construct validity of its tasks when evaluated against a GRS (p ≤ 0.003 in all cases). Regarding it’s automatically generated performance metrics, established outputs such as time taken (p ≤ 0.001) and instrument path length (p ≤ 0.007) also demonstrated good construct validity. However, two-thirds of the proposed ‘novel metrics’ the simulator reports could not distinguish participants based on arthroscopic experience. Face validity assessment rated the simulator as a realistic and useful tool for trainees, but the passive haptic feedback (a key feature of this simulator) is rated as less realistic.
The ArthroS simulator has good task construct validity based on established objective outputs, but some of the novel performance metrics could not distinguish between surgical experience. The passive haptic feedback of the simulator also needs improvement. If simulators could offer automated and validated performance feedback, this would facilitate improvements in the delivery of training by allowing trainees to practise and self-assess.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
Aggarwal R, Tully A, Grantcharov T, Larsen CR, Miskry T, Farthing A, Darzi A (2006) Virtual reality simulation training can improve technical skills during laparoscopic salpingectomy for ectopic pregnancy. Br J Obstet Gynaec 113:1382–1387
Akhtar K, Standfield NJ, Gupte CM, Tuijthof GJM (2015) Chapter 7: Virtual reality simulators. In: Karahan M, Kerkhoffs GMMJ, Randelli P, Tuijthof GJM (eds) Effective training in arthroscopic skills, 1st edn. Springer, Berlin, pp 71–80
Alvand A, Auplish S, Khan T, Gill HS, Rees JL (2011) Identifying orthopaedic surgeons of the future: the inability of some medical students to achieve competence in basic arthroscopic tasks despite training: a randomised study. J Bone Joint Surg Br 93:1586–1591
Alvand A, Khan T, Al-Ali S, Jackson WF, Price AJ, Rees JL (2012) Simple visual parameters for objective assessment of arthroscopic skill. J Bone Joint Surg Am 94:e97. doi:10.2106/JBJS.K.01437
Angelo RL, Ryu RKN, Pedowitz RA, Gallagher AG (2015) Metric development for an arthroscopic Bankart procedure: assessment of face and content validity. Arthroscopy 31:1430–1440
Frostick S, Baird E, Bale S, Banks T, Bhowal B, Kellett C, Cole A, Goodwin M, Hadfield-Law L, Hopgood P, Pitts D, Turner P, Reed M, Sher L, Tudor F (2013) Trauma and orthopaedic curriculum mapped to simulation options. British Orthopaedic Association, London. https://www.iscp.ac.uk/static/public/TO_Putting_simulation_into_practice.pdf
Brown D (2013) The role of simulation in the learning of surgical skills. Ann R Coll Surg Eng (Suppl) 95:292–295
Cannon WD, Nicandri GT, Reinig K, Mevis H, Wittstein J (2014) Evaluation of skill level between trainees and community orthopaedic surgeons using a virtual reality arthroscopic knee simulator. J Bone Joint Surg Am 96:e57. doi:10.2106/JBJS.M.00779
Coughlin RP, Pauyo T, Sutton JC, Coughlin LP, Bergeron SG (2015) A validated orthopaedic surgical simulation model for training and evaluation of basic arthroscopic skills. J Bone Joint Surg Am 97:1465–1471
Darzi A, Smith S, Taffinder N (1999) Assessing operative skill: needs to become more objective. Br Med J 318:887–888
Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193:479–485
Ferguson JY, Alvand A, Price AJ, Rees JL (2015) Chapter 8: Theory on simulator validation. In: Karahan M, Kerkhoffs GMMJ, Randelli P, Tuijthof GJM (eds) Effective training in arthroscopic skills, 1st edn. Springer, Berlin, pp 81–94
Fucentese SF, Rahm S, Wieser K, Spillmann J, Harders M, Koch PP (2015) Evaluation of a virtual-reality-based simulator using passive haptic feedback for knee arthroscopy. Knee Surg Sports Traumatol Arthrosc 23:1077–1085
Gélinas-Phaneuf N, Choudhury N, Al-Habib AR, Cabral A, Nadeau E, Mora V, Pazos V, Debergue P, DiRaddo R, Del Maestro RF (2014) Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg 9:1–9. doi:10.1007/s11548-013-0905-8
Hodgins JL, Veillette C (2013) Arthroscopic proficiency: methods in evaluating competency. BMC Med Educ 13:61
Howells NR, Brinsden MD, Gill RS, Carr AJ, Rees JL (2008) Motion analysis: a validated method for showing skill levels in arthroscopy. Arthroscopy 24:335–342
Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL (2008) Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br 90:494–499
Hunter S, McLaren P (1993) Specialist medical training and the Calman report. Br Med J 306:1281–1282
Jacobsen ME, Andersen MJ, Hansen CO, Konge L (2015) Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability. J Bone Joint Surg Am 97:775–781
Jamieson S (2004) Likert scales: how to (ab)use them. Med Educ 38:1217–1218
Koehler RJ, Amsdell S, Arendt EA, Bisson LJ, Bramen JP, Butler A, Cosgarea AJ, Harner CD, Garrett WE, Olson T, Warme WJ, Nicandri GT (2013) The arthroscopic surgical skill evaluation tool (ASSET). Am J Sports Med 41:1229–1237
Lopez G, Wright R, Martin D, Jung J, Bracey D, Gupta R (2015) A cost-effective junior resident training and assessment simulator for orthopaedic surgical skills via fundamentals of orthopaedic surgery: AAOS exhibit selection. J Bone Joint Surg Am 97:659–666
Marsh H (2015) Better not look down…. Ann R Coll Surg Eng (Suppl) 97:339–342
Milburn JA, Khera G, Hornby ST, Malone PSC, Fitzgerald JEF (2012) Introduction, availability and role of simulation in surgical education and training: review of current evidence and recommendations from the Association of Surgeons in Training. Int J Surg 10:393–398
Morrow G, Burford B, Carter M, Illing J (2012) The impact of the working time regulations on medical education and training: final report on primary research. General Medical Council, London. http://www.gmc-uk.org/The_Impact_of_the_Working_Time_Regulations_on_Medical_Education_and_Training___Final_Report_on_Primary_Research.pdf_51157039.pdf
Nousiainen MT, McQueen SA, Ferguson P, Alman B, Kraemer W, Safir O, Reznick R, Sonnadara R (2015) Simulation for teaching orthopaedic residents in a competency-based curriculum: do the benefits justify the increased costs? Clin Orthop Relat Res. doi:10.1007/s11999-015-4512-6
Palter VN, Grantcharov TP (2014) Individualized deliberate practice on a virtual reality simulator improves technical performance of surgical novices in the operating room. Ann Surg 259:443–448
Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ (2009) The role of haptic feedback in laparoscopic simulation training. J Surg Res 156:312–316
Pedowitz RA, Nicandri GT, Angelo RL, Ryu RKN, Gallagher AG (2015) Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester. Arthroscopy 31(10):1872–1879
Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern 13:257–266
Shah J, Darzi A (2002) Virtual reality flexible cystoscopy: a validation study. Br J Urol 90:828–832
Stunt JJ, Kerkhoffs GMMJ, Horeman T, van Dijk CN, Tuijthof GJM (2014) Validation of the PASSPORT V2 training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc. doi:10.1007/s00167-014-3213-0
Stunt JJ, Kerkhoffs GMMJ, van Dijk CN, Tuijthof GJM (2015) Validation of the ArthroS virtual reality simulator for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 23:3436–3442
Tashiro Y, Miura H, Nakanishi Y, Okazaki K, Iwamoto Y (2009) Evaluation of skills in arthroscopic training based on trajectory and force data. Clin Orthop Relat Res 467:546–552
Tuijthof GJM, van Sterkenburg MN, Sierevelt IN, van Oldenrijk J, Van Dijk CN, Kerkhoffs GMMJ (2010) First validation of the PASSPORT training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 18:218–224
VirtaMed AG (2014) VirtaMed ArthroS Virtual reality training simulator for knee and shoulder arthroscopy factsheet. VirtaMed, Zurich. http://www.virtamed.com/files/1914/2504/7801/VirtaMed_ArthroS_Factsheet_150227.pdf
Weiner IB, Craighead WE (eds) (2010) The corsini encyclopedia of psychology. Wiley, Hoboken
Wentink M, Stassen LPS, Alwayn I, Hosman RJAW, Stassen HG (2003) Rasmussen’s model of human behavior in laparoscopy training. Surg Endosc 17:1241–1246
Ziegler R, Fischer G, Müller W, Göbel M (1995) Virtual reality arthroscopy training simulator. Comput Biol Med 25:193–203
The ArthroS simulator used in this study was provided on loan from VirtaMed. The NIHR Oxford Musculoskeletal Biomedical Research Unit provided infrastructure support.
About this article
Cite this article
Garfjeld Roberts, P., Guyver, P., Baldwin, M. et al. Validation of the updated ArthroS simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics. Knee Surg Sports Traumatol Arthrosc 25, 616–625 (2017). https://doi.org/10.1007/s00167-016-4114-1
- Virtual reality