Skip to main content
Log in

Performance metrics in mastoidectomy training: a systematic review

  • Review Article
  • Published:
European Archives of Oto-Rhino-Laryngology Aims and scope Submit manuscript

Abstract

Objective

To investigate validity evidence, and strengths and limitations of performance metrics in mastoidectomy training.

Methods

A systematic review following the PRISMA guidelines. Studies reporting performance metrics in mastoidectomy/temporal bone surgery were included. Data on design, outcomes, and results were extracted by two reviewers. Validity evidence according to Messick’s framework and level of evidence were assessed.

Results

The search yielded a total of 1085 studies from the years 1947–2018 and 35 studies were included for full data extraction after abstract and full-text screening. 33 different metrics on mastoidectomy performance were identified and ranked according to the number of reports. Most of the 33 metrics identified had some amount of validity evidence. The metrics with most validity evidence were related to drilling time, volume drilled per time, force applied near vital structures, and volume removed.

Conclusions

This review provides an overview of current metrics of mastoidectomy performance, their validity, strengths and limitations, and identifies the gap in validity evidence of some metrics. Evidence-based metrics can be used for performance assessment in temporal bone surgery and for providing integrated and automated feedback in virtual reality simulation training. The use of such metrics in simulation-based mastoidectomy training can potentially address some of the limitations in current temporal bone skill assessment and ease assessment in repeated practice. However, at present, an automated feedback based on metrics in VR simulation does not have sufficient empirical basis and has not been generally accepted for use in training and certification.

Level of evidence

2a.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165(3):358–361

    Article  PubMed  CAS  Google Scholar 

  2. Cook DA (2014) How much evidence does it take? A cumulative meta-analysis of outcomes of simulation-based education. Medical education 48(8):750–760. https://doi.org/10.1111/medu.12473

    Article  PubMed  Google Scholar 

  3. Draycott TJ, Crofts JF, Ash JP, Wilson LV, Yard E, Sibanda T, Whitelaw A (2008) Improving neonatal outcome through practical shoulder dystocia training. Obstet Gynecol 112(1):14–20. https://doi.org/10.1097/AOG.0b013e31817bbc61

    Article  PubMed  Google Scholar 

  4. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB (2011) Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med 86(6):706–711. https://doi.org/10.1097/ACM.0b013e318217e119

    Article  PubMed  PubMed Central  Google Scholar 

  5. Vaughan N, Dubey VN, Wainwright TW, Middleton RG (2016) A review of virtual reality based training simulators for orthopaedic surgery. Med Eng Phys 38(2):59–71. https://doi.org/10.1016/j.medengphy.2015.11.021

    Article  PubMed  Google Scholar 

  6. Andersen SA, Foghsgaard S, Konge L, Caye-Thomasen P, Sorensen MS (2016) The effect of self-directed virtual reality simulation on dissection training performance in mastoidectomy. Laryngoscope 126(8):1883–1888. https://doi.org/10.1002/lary.25710

    Article  PubMed  Google Scholar 

  7. Fried MP, Sadoughi B, Gibber MJ, Jacobs JB, Lebowitz RA, Ross DA, Bent JP 3rd, Parikh SR, Sasaki CT, Schaefer SD (2010) From virtual reality to the operating room: the endoscopic sinus surgery simulator experiment. Otolaryngol Head Neck Surg 142(2):202–207. https://doi.org/10.1016/j.otohns.2009.11.023

    Article  PubMed  Google Scholar 

  8. Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, Erwin PJ, Hamstra SJ (2011) Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 306(9):978–988. https://doi.org/10.1001/jama.2011.1234

    Article  PubMed  CAS  Google Scholar 

  9. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ (2005) Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teacher 27(1):10–28. https://doi.org/10.1080/01421590500046924

    Article  Google Scholar 

  10. Sachdeva AK, Buyske J, Dunnington GL, Sanfey HA, Mellinger JD, Scott DJ, Satava R, Fried GM, Jacobs LM, Burns KJ (2011) A new paradigm for surgical procedural training. Curr Probl Surg 48(12):854–968. https://doi.org/10.1067/j.cpsurg.2011.08.003

    Article  PubMed  Google Scholar 

  11. Frithioff A, Sorensen MS, Andersen SAW (2018) European status on temporal bone training: a questionnaire study. Eur Arch Oto-Rhino-Laryngol 275(2):357–363. https://doi.org/10.1007/s00405-017-4824-0

    Article  Google Scholar 

  12. Bhutta MF (2016) A review of simulation platforms in surgery of the temporal bone. Clin Otolaryngol 41 (5):539–545. https://doi.org/10.1111/coa.12560

    Article  PubMed  CAS  Google Scholar 

  13. Arora A, Hall A, Kotecha J, Burgess C, Khemani S, Darzi A, Singh A, Tolley N (2015) Virtual reality simulation training in temporal bone surgery. Clin Otolaryngol 40(2):153–159. https://doi.org/10.1111/coa.12352

    Article  PubMed  CAS  Google Scholar 

  14. Sethia R, Kerwin TF, Wiet GJ (2016) Performance assessment for mastoidectomy: state of the art review. Otolaryngol Head Neck Surg. https://doi.org/10.1177/0194599816670886

    Article  PubMed  PubMed Central  Google Scholar 

  15. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151(4):264–269, W264

    Article  PubMed  Google Scholar 

  16. Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I, Farrell TM (2015) Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 261(2):251–262. https://doi.org/10.1097/SLA.0000000000000520

    Article  PubMed  Google Scholar 

  17. Khemani S, Arora A, Singh A, Tolley N, Darzi A (2012) Objective skills assessment and construct validation of a virtual reality temporal bone simulator. Otology and neurotology: official publication of the American Otological Society. Am Neurotol Soc Eur Acad Otol Neurotol 33(7):1225–1231. https://doi.org/10.1097/MAO.0b013e31825e7977

    Article  Google Scholar 

  18. Zhao YC, Kennedy G, Hall R, O’Leary S (2010) Differentiating levels of surgical experience on a virtual reality temporal bone simulator. Otolaryngol Head Neck Surg 143(5 Suppl 3):S30–S35. https://doi.org/10.1016/j.otohns.2010.03.008

    Article  PubMed  Google Scholar 

  19. Ioannou I, Avery A, Zhou Y, Szudek J, Kennedy G, O’Leary S (2014) The effect of fidelity: how expert behavior changes in a virtual reality environment. Laryngoscope 124(9):2144–2150. https://doi.org/10.1002/lary.24708

    Article  PubMed  Google Scholar 

  20. Linke R, Leichtle A, Sheikh F, Schmidt C, Frenzel H, Graefe H, Wollenberg B, Meyer JE (2013) Assessment of skills using a virtual reality temporal bone surgery simulator. Acta otorhinolaryngologica Italica: organo ufficiale della Societa italiana di otorinolaringologia e chirurgia cervico-facciale 33 (4):273–281

  21. Morris D, Sewell C, Barbagli F, Salisbury K, Blevins NH, Girod S (2006) Visuohaptic simulation of bone surgery for training and evaluation. IEEE Comput Graph Appl 26(6):48–57

    Article  PubMed  Google Scholar 

  22. Sewell C, Morris D, Blevins NH, Dutta S, Agrawal S, Barbagli F, Salisbury K (2008) Providing metrics and performance feedback in a surgical simulator. Comput Aid Surg 13(2):63–81. https://doi.org/10.3109/10929080801957712

    Article  Google Scholar 

  23. Satava RM, Cuschieri A, Hamdorf J, Metrics for Objective Assessment of Surgical Skills W (2003) Metrics for objective assessment. Surg Endosc 17(2):220–226. https://doi.org/10.1007/s00464-002-8869-8

    Article  PubMed  CAS  Google Scholar 

  24. Konge L, Annema J, Clementsen P, Minddal V, Vilmann P, Ringsted C (2013) Using virtual-reality simulation to assess performance in endobronchial ultrasound. Respir Int Rev Thorac Dis 86(1):59–65. https://doi.org/10.1159/000350428

    Article  Google Scholar 

  25. Kerwin T, Wiet G, Stredney D, Shen H-W (2012) Automatic scoring of virtual mastoidectomies using expert examples. Int J Comput Assist Radiol Surg 7(1):1–11. https://doi.org/10.1007/s11548-011-0566-4

    Article  PubMed  Google Scholar 

Download references

Funding

None.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Fahd Al-Shahrestani.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Shahrestani, F., Sørensen, M.S. & Andersen, S.A.W. Performance metrics in mastoidectomy training: a systematic review. Eur Arch Otorhinolaryngol 276, 657–664 (2019). https://doi.org/10.1007/s00405-018-05265-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00405-018-05265-9

Keywords

Navigation