Knee Surgery, Sports Traumatology, Arthroscopy

, Volume 25, Issue 2, pp 616–625 | Cite as

Validation of the updated ArthroS simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics

  • Patrick Garfjeld Roberts
  • Paul Guyver
  • Mathew Baldwin
  • Kash Akhtar
  • Abtin Alvand
  • Andrew J. Price
  • Jonathan L. Rees
Sports Traumatology

Abstract

Purpose

To assess the construct and face validity of ArthroS, a passive haptic VR simulator. A secondary aim was to evaluate the novel performance metrics produced by this simulator.

Methods

Two groups of 30 participants, each divided into novice, intermediate or expert based on arthroscopic experience, completed three separate tasks on either the knee or shoulder module of the simulator. Performance was recorded using 12 automatically generated performance metrics and video footage of the arthroscopic procedures. The videos were blindly assessed using a validated global rating scale (GRS). Participants completed a survey about the simulator’s realism and training utility.

Results

This new simulator demonstrated construct validity of its tasks when evaluated against a GRS (p ≤ 0.003 in all cases). Regarding it’s automatically generated performance metrics, established outputs such as time taken (p ≤ 0.001) and instrument path length (p ≤ 0.007) also demonstrated good construct validity. However, two-thirds of the proposed ‘novel metrics’ the simulator reports could not distinguish participants based on arthroscopic experience. Face validity assessment rated the simulator as a realistic and useful tool for trainees, but the passive haptic feedback (a key feature of this simulator) is rated as less realistic.

Conclusion

The ArthroS simulator has good task construct validity based on established objective outputs, but some of the novel performance metrics could not distinguish between surgical experience. The passive haptic feedback of the simulator also needs improvement. If simulators could offer automated and validated performance feedback, this would facilitate improvements in the delivery of training by allowing trainees to practise and self-assess.

Keywords

Simulation Arthroscopy Training Virtual reality 

References

  1. 1.
    Aggarwal R, Tully A, Grantcharov T, Larsen CR, Miskry T, Farthing A, Darzi A (2006) Virtual reality simulation training can improve technical skills during laparoscopic salpingectomy for ectopic pregnancy. Br J Obstet Gynaec 113:1382–1387CrossRefGoogle Scholar
  2. 2.
    Akhtar K, Standfield NJ, Gupte CM, Tuijthof GJM (2015) Chapter 7: Virtual reality simulators. In: Karahan M, Kerkhoffs GMMJ, Randelli P, Tuijthof GJM (eds) Effective training in arthroscopic skills, 1st edn. Springer, Berlin, pp 71–80Google Scholar
  3. 3.
    Alvand A, Auplish S, Khan T, Gill HS, Rees JL (2011) Identifying orthopaedic surgeons of the future: the inability of some medical students to achieve competence in basic arthroscopic tasks despite training: a randomised study. J Bone Joint Surg Br 93:1586–1591CrossRefPubMedGoogle Scholar
  4. 4.
    Alvand A, Khan T, Al-Ali S, Jackson WF, Price AJ, Rees JL (2012) Simple visual parameters for objective assessment of arthroscopic skill. J Bone Joint Surg Am 94:e97. doi:10.2106/JBJS.K.01437 CrossRefPubMedGoogle Scholar
  5. 5.
    Angelo RL, Ryu RKN, Pedowitz RA, Gallagher AG (2015) Metric development for an arthroscopic Bankart procedure: assessment of face and content validity. Arthroscopy 31:1430–1440CrossRefPubMedGoogle Scholar
  6. 6.
    Frostick S, Baird E, Bale S, Banks T, Bhowal B, Kellett C, Cole A, Goodwin M, Hadfield-Law L, Hopgood P, Pitts D, Turner P, Reed M, Sher L, Tudor F (2013) Trauma and orthopaedic curriculum mapped to simulation options. British Orthopaedic Association, London. https://www.iscp.ac.uk/static/public/TO_Putting_simulation_into_practice.pdf
  7. 7.
    Brown D (2013) The role of simulation in the learning of surgical skills. Ann R Coll Surg Eng (Suppl) 95:292–295CrossRefGoogle Scholar
  8. 8.
    Cannon WD, Nicandri GT, Reinig K, Mevis H, Wittstein J (2014) Evaluation of skill level between trainees and community orthopaedic surgeons using a virtual reality arthroscopic knee simulator. J Bone Joint Surg Am 96:e57. doi:10.2106/JBJS.M.00779 CrossRefPubMedGoogle Scholar
  9. 9.
    Coughlin RP, Pauyo T, Sutton JC, Coughlin LP, Bergeron SG (2015) A validated orthopaedic surgical simulation model for training and evaluation of basic arthroscopic skills. J Bone Joint Surg Am 97:1465–1471CrossRefPubMedGoogle Scholar
  10. 10.
    Darzi A, Smith S, Taffinder N (1999) Assessing operative skill: needs to become more objective. Br Med J 318:887–888CrossRefGoogle Scholar
  11. 11.
    Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193:479–485CrossRefPubMedGoogle Scholar
  12. 12.
    Ferguson JY, Alvand A, Price AJ, Rees JL (2015) Chapter 8: Theory on simulator validation. In: Karahan M, Kerkhoffs GMMJ, Randelli P, Tuijthof GJM (eds) Effective training in arthroscopic skills, 1st edn. Springer, Berlin, pp 81–94Google Scholar
  13. 13.
    Fucentese SF, Rahm S, Wieser K, Spillmann J, Harders M, Koch PP (2015) Evaluation of a virtual-reality-based simulator using passive haptic feedback for knee arthroscopy. Knee Surg Sports Traumatol Arthrosc 23:1077–1085CrossRefPubMedGoogle Scholar
  14. 14.
    Gélinas-Phaneuf N, Choudhury N, Al-Habib AR, Cabral A, Nadeau E, Mora V, Pazos V, Debergue P, DiRaddo R, Del Maestro RF (2014) Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg 9:1–9. doi:10.1007/s11548-013-0905-8 CrossRefPubMedGoogle Scholar
  15. 15.
    Hodgins JL, Veillette C (2013) Arthroscopic proficiency: methods in evaluating competency. BMC Med Educ 13:61CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Howells NR, Brinsden MD, Gill RS, Carr AJ, Rees JL (2008) Motion analysis: a validated method for showing skill levels in arthroscopy. Arthroscopy 24:335–342CrossRefPubMedGoogle Scholar
  17. 17.
    Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL (2008) Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br 90:494–499CrossRefPubMedGoogle Scholar
  18. 18.
    Hunter S, McLaren P (1993) Specialist medical training and the Calman report. Br Med J 306:1281–1282CrossRefGoogle Scholar
  19. 19.
    Jacobsen ME, Andersen MJ, Hansen CO, Konge L (2015) Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability. J Bone Joint Surg Am 97:775–781CrossRefPubMedGoogle Scholar
  20. 20.
    Jamieson S (2004) Likert scales: how to (ab)use them. Med Educ 38:1217–1218CrossRefPubMedGoogle Scholar
  21. 21.
    Koehler RJ, Amsdell S, Arendt EA, Bisson LJ, Bramen JP, Butler A, Cosgarea AJ, Harner CD, Garrett WE, Olson T, Warme WJ, Nicandri GT (2013) The arthroscopic surgical skill evaluation tool (ASSET). Am J Sports Med 41:1229–1237CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Lopez G, Wright R, Martin D, Jung J, Bracey D, Gupta R (2015) A cost-effective junior resident training and assessment simulator for orthopaedic surgical skills via fundamentals of orthopaedic surgery: AAOS exhibit selection. J Bone Joint Surg Am 97:659–666CrossRefPubMedGoogle Scholar
  23. 23.
    Marsh H (2015) Better not look down…. Ann R Coll Surg Eng (Suppl) 97:339–342CrossRefGoogle Scholar
  24. 24.
    Milburn JA, Khera G, Hornby ST, Malone PSC, Fitzgerald JEF (2012) Introduction, availability and role of simulation in surgical education and training: review of current evidence and recommendations from the Association of Surgeons in Training. Int J Surg 10:393–398CrossRefPubMedGoogle Scholar
  25. 25.
    Morrow G, Burford B, Carter M, Illing J (2012) The impact of the working time regulations on medical education and training: final report on primary research. General Medical Council, London. http://www.gmc-uk.org/The_Impact_of_the_Working_Time_Regulations_on_Medical_Education_and_Training___Final_Report_on_Primary_Research.pdf_51157039.pdf
  26. 26.
    Nousiainen MT, McQueen SA, Ferguson P, Alman B, Kraemer W, Safir O, Reznick R, Sonnadara R (2015) Simulation for teaching orthopaedic residents in a competency-based curriculum: do the benefits justify the increased costs? Clin Orthop Relat Res. doi:10.1007/s11999-015-4512-6 Google Scholar
  27. 27.
    Palter VN, Grantcharov TP (2014) Individualized deliberate practice on a virtual reality simulator improves technical performance of surgical novices in the operating room. Ann Surg 259:443–448CrossRefPubMedGoogle Scholar
  28. 28.
    Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ (2009) The role of haptic feedback in laparoscopic simulation training. J Surg Res 156:312–316CrossRefPubMedGoogle Scholar
  29. 29.
    Pedowitz RA, Nicandri GT, Angelo RL, Ryu RKN, Gallagher AG (2015) Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester. Arthroscopy 31(10):1872–1879CrossRefPubMedGoogle Scholar
  30. 30.
    Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern 13:257–266CrossRefGoogle Scholar
  31. 31.
    Shah J, Darzi A (2002) Virtual reality flexible cystoscopy: a validation study. Br J Urol 90:828–832CrossRefGoogle Scholar
  32. 32.
    Stunt JJ, Kerkhoffs GMMJ, Horeman T, van Dijk CN, Tuijthof GJM (2014) Validation of the PASSPORT V2 training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc. doi:10.1007/s00167-014-3213-0 Google Scholar
  33. 33.
    Stunt JJ, Kerkhoffs GMMJ, van Dijk CN, Tuijthof GJM (2015) Validation of the ArthroS virtual reality simulator for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 23:3436–3442CrossRefPubMedGoogle Scholar
  34. 34.
    Tashiro Y, Miura H, Nakanishi Y, Okazaki K, Iwamoto Y (2009) Evaluation of skills in arthroscopic training based on trajectory and force data. Clin Orthop Relat Res 467:546–552CrossRefPubMedGoogle Scholar
  35. 35.
    Tuijthof GJM, van Sterkenburg MN, Sierevelt IN, van Oldenrijk J, Van Dijk CN, Kerkhoffs GMMJ (2010) First validation of the PASSPORT training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 18:218–224CrossRefPubMedGoogle Scholar
  36. 36.
    VirtaMed AG (2014) VirtaMed ArthroS Virtual reality training simulator for knee and shoulder arthroscopy factsheet. VirtaMed, Zurich. http://www.virtamed.com/files/1914/2504/7801/VirtaMed_ArthroS_Factsheet_150227.pdf
  37. 37.
    Weiner IB, Craighead WE (eds) (2010) The corsini encyclopedia of psychology. Wiley, HobokenGoogle Scholar
  38. 38.
    Wentink M, Stassen LPS, Alwayn I, Hosman RJAW, Stassen HG (2003) Rasmussen’s model of human behavior in laparoscopy training. Surg Endosc 17:1241–1246CrossRefPubMedGoogle Scholar
  39. 39.
    Ziegler R, Fischer G, Müller W, Göbel M (1995) Virtual reality arthroscopy training simulator. Comput Biol Med 25:193–203CrossRefPubMedGoogle Scholar

Copyright information

© European Society of Sports Traumatology, Knee Surgery, Arthroscopy (ESSKA) 2016

Authors and Affiliations

  1. 1.Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal SciencesUniversity of OxfordOxfordUK
  2. 2.MDHU DerrifordPlymouth Hospitals NHS TrustOxfordUK
  3. 3.Health Education Thames ValleyOxfordUK
  4. 4.The Blizard Institute, Barts and The London School of Medicine and DentistryQueen Mary University of LondonLondonUK

Personalised recommendations