Skip to main content
SpringerLink
Log in

Validation of the updated ArthroS simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics

  • Sports Traumatology
  • Published:
Knee Surgery, Sports Traumatology, Arthroscopy Aims and scope

Abstract

Purpose

To assess the construct and face validity of ArthroS, a passive haptic VR simulator. A secondary aim was to evaluate the novel performance metrics produced by this simulator.

Methods

Two groups of 30 participants, each divided into novice, intermediate or expert based on arthroscopic experience, completed three separate tasks on either the knee or shoulder module of the simulator. Performance was recorded using 12 automatically generated performance metrics and video footage of the arthroscopic procedures. The videos were blindly assessed using a validated global rating scale (GRS). Participants completed a survey about the simulator’s realism and training utility.

Results

This new simulator demonstrated construct validity of its tasks when evaluated against a GRS (p ≤ 0.003 in all cases). Regarding it’s automatically generated performance metrics, established outputs such as time taken (p ≤ 0.001) and instrument path length (p ≤ 0.007) also demonstrated good construct validity. However, two-thirds of the proposed ‘novel metrics’ the simulator reports could not distinguish participants based on arthroscopic experience. Face validity assessment rated the simulator as a realistic and useful tool for trainees, but the passive haptic feedback (a key feature of this simulator) is rated as less realistic.

Conclusion

The ArthroS simulator has good task construct validity based on established objective outputs, but some of the novel performance metrics could not distinguish between surgical experience. The passive haptic feedback of the simulator also needs improvement. If simulators could offer automated and validated performance feedback, this would facilitate improvements in the delivery of training by allowing trainees to practise and self-assess.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Aggarwal R, Tully A, Grantcharov T, Larsen CR, Miskry T, Farthing A, Darzi A (2006) Virtual reality simulation training can improve technical skills during laparoscopic salpingectomy for ectopic pregnancy. Br J Obstet Gynaec 113:1382–1387

    Article  CAS  Google Scholar 

  2. Akhtar K, Standfield NJ, Gupte CM, Tuijthof GJM (2015) Chapter 7: Virtual reality simulators. In: Karahan M, Kerkhoffs GMMJ, Randelli P, Tuijthof GJM (eds) Effective training in arthroscopic skills, 1st edn. Springer, Berlin, pp 71–80

    Google Scholar 

  3. Alvand A, Auplish S, Khan T, Gill HS, Rees JL (2011) Identifying orthopaedic surgeons of the future: the inability of some medical students to achieve competence in basic arthroscopic tasks despite training: a randomised study. J Bone Joint Surg Br 93:1586–1591

    Article  CAS  PubMed  Google Scholar 

  4. Alvand A, Khan T, Al-Ali S, Jackson WF, Price AJ, Rees JL (2012) Simple visual parameters for objective assessment of arthroscopic skill. J Bone Joint Surg Am 94:e97. doi:10.2106/JBJS.K.01437

    Article  CAS  PubMed  Google Scholar 

  5. Angelo RL, Ryu RKN, Pedowitz RA, Gallagher AG (2015) Metric development for an arthroscopic Bankart procedure: assessment of face and content validity. Arthroscopy 31:1430–1440

    Article  PubMed  Google Scholar 

  6. Frostick S, Baird E, Bale S, Banks T, Bhowal B, Kellett C, Cole A, Goodwin M, Hadfield-Law L, Hopgood P, Pitts D, Turner P, Reed M, Sher L, Tudor F (2013) Trauma and orthopaedic curriculum mapped to simulation options. British Orthopaedic Association, London. https://www.iscp.ac.uk/static/public/TO_Putting_simulation_into_practice.pdf

  7. Brown D (2013) The role of simulation in the learning of surgical skills. Ann R Coll Surg Eng (Suppl) 95:292–295

    Article  Google Scholar 

  8. Cannon WD, Nicandri GT, Reinig K, Mevis H, Wittstein J (2014) Evaluation of skill level between trainees and community orthopaedic surgeons using a virtual reality arthroscopic knee simulator. J Bone Joint Surg Am 96:e57. doi:10.2106/JBJS.M.00779

    Article  PubMed  Google Scholar 

  9. Coughlin RP, Pauyo T, Sutton JC, Coughlin LP, Bergeron SG (2015) A validated orthopaedic surgical simulation model for training and evaluation of basic arthroscopic skills. J Bone Joint Surg Am 97:1465–1471

    Article  PubMed  Google Scholar 

  10. Darzi A, Smith S, Taffinder N (1999) Assessing operative skill: needs to become more objective. Br Med J 318:887–888

    Article  CAS  Google Scholar 

  11. Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193:479–485

    Article  CAS  PubMed  Google Scholar 

  12. Ferguson JY, Alvand A, Price AJ, Rees JL (2015) Chapter 8: Theory on simulator validation. In: Karahan M, Kerkhoffs GMMJ, Randelli P, Tuijthof GJM (eds) Effective training in arthroscopic skills, 1st edn. Springer, Berlin, pp 81–94

    Google Scholar 

  13. Fucentese SF, Rahm S, Wieser K, Spillmann J, Harders M, Koch PP (2015) Evaluation of a virtual-reality-based simulator using passive haptic feedback for knee arthroscopy. Knee Surg Sports Traumatol Arthrosc 23:1077–1085

    Article  PubMed  Google Scholar 

  14. Gélinas-Phaneuf N, Choudhury N, Al-Habib AR, Cabral A, Nadeau E, Mora V, Pazos V, Debergue P, DiRaddo R, Del Maestro RF (2014) Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg 9:1–9. doi:10.1007/s11548-013-0905-8

    Article  PubMed  Google Scholar 

  15. Hodgins JL, Veillette C (2013) Arthroscopic proficiency: methods in evaluating competency. BMC Med Educ 13:61

    Article  PubMed  PubMed Central  Google Scholar 

  16. Howells NR, Brinsden MD, Gill RS, Carr AJ, Rees JL (2008) Motion analysis: a validated method for showing skill levels in arthroscopy. Arthroscopy 24:335–342

    Article  PubMed  Google Scholar 

  17. Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL (2008) Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br 90:494–499

    Article  CAS  PubMed  Google Scholar 

  18. Hunter S, McLaren P (1993) Specialist medical training and the Calman report. Br Med J 306:1281–1282

    Article  CAS  Google Scholar 

  19. Jacobsen ME, Andersen MJ, Hansen CO, Konge L (2015) Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability. J Bone Joint Surg Am 97:775–781

    Article  PubMed  Google Scholar 

  20. Jamieson S (2004) Likert scales: how to (ab)use them. Med Educ 38:1217–1218

    Article  PubMed  Google Scholar 

  21. Koehler RJ, Amsdell S, Arendt EA, Bisson LJ, Bramen JP, Butler A, Cosgarea AJ, Harner CD, Garrett WE, Olson T, Warme WJ, Nicandri GT (2013) The arthroscopic surgical skill evaluation tool (ASSET). Am J Sports Med 41:1229–1237

    Article  PubMed  PubMed Central  Google Scholar 

  22. Lopez G, Wright R, Martin D, Jung J, Bracey D, Gupta R (2015) A cost-effective junior resident training and assessment simulator for orthopaedic surgical skills via fundamentals of orthopaedic surgery: AAOS exhibit selection. J Bone Joint Surg Am 97:659–666

    Article  PubMed  Google Scholar 

  23. Marsh H (2015) Better not look down…. Ann R Coll Surg Eng (Suppl) 97:339–342

    Article  Google Scholar 

  24. Milburn JA, Khera G, Hornby ST, Malone PSC, Fitzgerald JEF (2012) Introduction, availability and role of simulation in surgical education and training: review of current evidence and recommendations from the Association of Surgeons in Training. Int J Surg 10:393–398

    Article  CAS  PubMed  Google Scholar 

  25. Morrow G, Burford B, Carter M, Illing J (2012) The impact of the working time regulations on medical education and training: final report on primary research. General Medical Council, London. http://www.gmc-uk.org/The_Impact_of_the_Working_Time_Regulations_on_Medical_Education_and_Training___Final_Report_on_Primary_Research.pdf_51157039.pdf

  26. Nousiainen MT, McQueen SA, Ferguson P, Alman B, Kraemer W, Safir O, Reznick R, Sonnadara R (2015) Simulation for teaching orthopaedic residents in a competency-based curriculum: do the benefits justify the increased costs? Clin Orthop Relat Res. doi:10.1007/s11999-015-4512-6

    Google Scholar 

  27. Palter VN, Grantcharov TP (2014) Individualized deliberate practice on a virtual reality simulator improves technical performance of surgical novices in the operating room. Ann Surg 259:443–448

    Article  PubMed  Google Scholar 

  28. Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ (2009) The role of haptic feedback in laparoscopic simulation training. J Surg Res 156:312–316

    Article  PubMed  Google Scholar 

  29. Pedowitz RA, Nicandri GT, Angelo RL, Ryu RKN, Gallagher AG (2015) Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester. Arthroscopy 31(10):1872–1879

    Article  PubMed  Google Scholar 

  30. Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern 13:257–266

    Article  Google Scholar 

  31. Shah J, Darzi A (2002) Virtual reality flexible cystoscopy: a validation study. Br J Urol 90:828–832

    Article  CAS  Google Scholar 

  32. Stunt JJ, Kerkhoffs GMMJ, Horeman T, van Dijk CN, Tuijthof GJM (2014) Validation of the PASSPORT V2 training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc. doi:10.1007/s00167-014-3213-0

    Google Scholar 

  33. Stunt JJ, Kerkhoffs GMMJ, van Dijk CN, Tuijthof GJM (2015) Validation of the ArthroS virtual reality simulator for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 23:3436–3442

    Article  CAS  PubMed  Google Scholar 

  34. Tashiro Y, Miura H, Nakanishi Y, Okazaki K, Iwamoto Y (2009) Evaluation of skills in arthroscopic training based on trajectory and force data. Clin Orthop Relat Res 467:546–552

    Article  PubMed  Google Scholar 

  35. Tuijthof GJM, van Sterkenburg MN, Sierevelt IN, van Oldenrijk J, Van Dijk CN, Kerkhoffs GMMJ (2010) First validation of the PASSPORT training environment for arthroscopic skills. Knee Surg Sports Traumatol Arthrosc 18:218–224

    Article  PubMed  Google Scholar 

  36. VirtaMed AG (2014) VirtaMed ArthroS Virtual reality training simulator for knee and shoulder arthroscopy factsheet. VirtaMed, Zurich. http://www.virtamed.com/files/1914/2504/7801/VirtaMed_ArthroS_Factsheet_150227.pdf

  37. Weiner IB, Craighead WE (eds) (2010) The corsini encyclopedia of psychology. Wiley, Hoboken

    Google Scholar 

  38. Wentink M, Stassen LPS, Alwayn I, Hosman RJAW, Stassen HG (2003) Rasmussen’s model of human behavior in laparoscopy training. Surg Endosc 17:1241–1246

    Article  CAS  PubMed  Google Scholar 

  39. Ziegler R, Fischer G, Müller W, Göbel M (1995) Virtual reality arthroscopy training simulator. Comput Biol Med 25:193–203

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

The ArthroS simulator used in this study was provided on loan from VirtaMed. The NIHR Oxford Musculoskeletal Biomedical Research Unit provided infrastructure support.

Author information

Authors and Affiliations

  1. Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK

    Patrick Garfjeld Roberts, Abtin Alvand, Andrew J. Price & Jonathan L. Rees

  2. MDHU Derriford, Plymouth Hospitals NHS Trust, Oxford, UK

    Paul Guyver

  3. Health Education Thames Valley, Oxford, UK

    Mathew Baldwin

  4. The Blizard Institute, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, London, UK

    Kash Akhtar

Authors
  1. Patrick Garfjeld Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Guyver
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mathew Baldwin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kash Akhtar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abtin Alvand
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andrew J. Price
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jonathan L. Rees
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick Garfjeld Roberts.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garfjeld Roberts, P., Guyver, P., Baldwin, M. et al. Validation of the updated ArthroS simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics. Knee Surg Sports Traumatol Arthrosc 25, 616–625 (2017). https://doi.org/10.1007/s00167-016-4114-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00167-016-4114-1

Keywords

Search

Navigation