Skip to main content

Content validity evidence for a simulation-based test of handheld otoscopy skills

Abstract

Purpose

At graduation from medical school, competency in otoscopy is often insufficient. Simulation-based training can be used to improve technical skills, but the suitability of the training model and assessment must be supported by validity evidence. The purpose of this study was to collect content validity evidence for a simulation-based test of handheld otoscopy skills.

Methods

First, a three-round Delphi study was conducted with a panel of nine clinical teachers in otorhinolaryngology (ORL) to determine the content requirements in our educational context. Next, the authenticity of relevant cases in a commercially available technology-enhanced simulator (Earsi, VR Magic, Germany) was evaluated by specialists in ORL. Finally, an integrated course was developed for the simulator based on these results.

Results

The Delphi study resulted in nine essential diagnoses of normal variations and pathologies that all junior doctors should be able to diagnose with a handheld otoscope. Twelve out of 15 tested simulator cases were correctly recognized by at least one ORL specialist. Fifteen cases from the simulator case library matched the essential diagnoses determined by the Delphi study and were integrated into the course.

Conclusion

Content validity evidence for a simulation-based test of handheld otoscopy skills was collected. This informed a simulation-based course that can be used for undergraduate training. The course needs to be further investigated in relation to other aspects of validity and for future self-directed training.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Availability of data and materials

Data is available from the authors upon request.

References

  1. 1.

    Jones WS (2006) Video otoscopy: bringing otoscopy out of the “black box”. Int J Pediatr Otorhinolaryngol 70:1875–1883. https://doi.org/10.1016/j.ijporl.2006.06.014

    Article  PubMed  Google Scholar 

  2. 2.

    Ringsted CV, Pallisgaard J, Falck G (2002) Physicians’ clinical skills after finishing internship. Ugeskr Læg 164:3211–3215

    PubMed  Google Scholar 

  3. 3.

    Oyewumi M, Brandt MG, Carrillo B et al (2016) Objective evaluation of otoscopy skills among family and community medicine, pediatric, and otolaryngology residents. J Surg Educ 73:129–135. https://doi.org/10.1016/j.jsurg.2015.07.011

    Article  PubMed  Google Scholar 

  4. 4.

    Fisher EW, Pfleiderer AG (1992) Assessment of the otoscopic skills of general practitioners and medical students: is there room for improvement? Br J Gen Pract 42:65–67

    CAS  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Morris E, Kesser BW, Peirce-Cottler S, Keeley M (2012) Development and validation of a novel ear simulator to teach pneumatic otoscopy. Simul Healthc 7:22–26. https://doi.org/10.1097/SIH.0b013e31822eac39

    Article  PubMed  Google Scholar 

  6. 6.

    Kyoto Kagakuku “Ear” Examination simulator 2. https://www.kyotokagaku.com/products/detail01/mw12.html. Accessed 19 May 2020

  7. 7.

    Wickens B, Lewis J, Morris DP et al (2015) Face and content validity of a novel, web-based otoscopy simulator for medical education. J Otolaryngol Head Neck Surg 44:7. https://doi.org/10.1186/s40463-015-0060-z

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Stepniak C, Wickens B, Husein M et al (2017) Blinded randomized controlled study of a web-based otoscopy simulator in undergraduate medical education. Laryngoscope 127:1306–1311. https://doi.org/10.1002/lary.26246

    Article  PubMed  Google Scholar 

  9. 9.

    Xu J, Campisi P, Forte V et al (2018) Effectiveness of discovery learning using a mobile otoscopy simulator on knowledge acquisition and retention in medical students: a randomized controlled trial. J Otolaryngol Head Neck Surg 47:1–8. https://doi.org/10.1186/s40463-018-0317-4

    Article  Google Scholar 

  10. 10.

    Lee DJ, Fu TS, Carrillo B et al (2015) Evaluation of an otoscopy simulator to teach otoscopy and normative anatomy to first year medical students. Laryngoscope 125:2159–2162. https://doi.org/10.1002/lary.25135

    Article  PubMed  Google Scholar 

  11. 11.

    Davies J, Djelic L, Campisi P et al (2014) Otoscopy simulation training in a classroom setting: a novel approach to teaching otoscopy to medical students. Laryngoscope 124:2594–2597. https://doi.org/10.1002/lary.24682

    Article  PubMed  Google Scholar 

  12. 12.

    Wu V, Sattar J, Cheon S, Beyea JA (2018) Ear disease knowledge and otoscopy skills transfer to real patients: a randomized controlled trial. J Surg Educ 75:1062–1069. https://doi.org/10.1016/j.jsurg.2017.12.011

    Article  PubMed  Google Scholar 

  13. 13.

    Dell'Era V, Garzaro M, Carenzo L et al (2020) An innovative and safe way to train novice ear nose and throat residents through simulation: the simorl experience. Acta Otorhinolaryngol Ital 40:19–25. https://doi.org/10.14639/0392-100X-N0128

    Article  Google Scholar 

  14. 14.

    Wu V, Beyea JA (2017) Evaluation of a web-based module and an otoscopy simulator in teaching ear disease. Otolaryngol Head Neck Surg 156:272–277. https://doi.org/10.1177/0194599816677697

    Article  PubMed  Google Scholar 

  15. 15.

    VRmagic Earsi Otoscope. https://www.vrmagic.com/medical-simulators/earsi-otoscope. Accessed 18 May 2020

  16. 16.

    Wiet GJ, Sørensen MS, Andersen SAW (2017) Otologic skills training. Otolaryngol Clin N Am 50:933–945. https://doi.org/10.1016/j.otc.2017.05.005

    Article  Google Scholar 

  17. 17.

    Saks K, Leijen Ä (2014) Distinguishing self-directed and self-regulated learning and measuring them in the E-learning context. Procedia Soc Behav Sci 112:190–198. https://doi.org/10.1016/j.sbspro.2014.01.1155

    Article  Google Scholar 

  18. 18.

    Yudkowsky R, Park YS, Lineberry M et al (2015) Setting mastery learning standards. Acad Med 90:1495–1500. https://doi.org/10.1097/ACM.0000000000000887

    Article  PubMed  Google Scholar 

  19. 19.

    Cook DA, Beckman TJ (2006) Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 119:166.e7–166.e16. https://doi.org/10.1016/j.amjmed.2005.10.036

    Article  Google Scholar 

  20. 20.

    Cook DA, Hatala R (2016) Validation of educational assessments: a primer for simulation and beyond. Adv Simul 1:31. https://doi.org/10.1186/s41077-016-0033-y

    Article  Google Scholar 

  21. 21.

    Borgersen NJ, Naur TMH, Sørensen SMD et al (2018) Gathering validity evidence for surgical simulation: a systematic review. Ann Surg 267:1063–1068. https://doi.org/10.1097/SLA.0000000000002652

    Article  PubMed  Google Scholar 

  22. 22.

    Hsu CC, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12:Article 10. https://doi.org/10.7275/pdz9-th90

  23. 23.

    Guldager MJ, Melchiors J, Andersen SAW (2020) Development and validation of an assessment tool for technical skills in handheld otoscopy. Ann Otol Rhinol Laryngol 129:715–721. https://doi.org/10.1177/0003489420904734

  24. 24.

    Downing SM (2006) Face validity of assessments: Faith-based interpretations or evidence-based science? Med Educ 40:7–8. https://doi.org/10.1111/j.1365-2929.2005.02361.x

    Article  PubMed  Google Scholar 

  25. 25.

    McMillan SS, King M, Tully MP (2016) How to use the nominal group and Delphi techniques. Int J Clin Pharm 38:655–662. https://doi.org/10.1007/s11096-016-0257-x

    Article  PubMed  PubMed Central  Google Scholar 

  26. 26.

    Waggoner J, Carline JD, Durning SJ (2016) Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad Med 91:663–668. https://doi.org/10.1097/ACM.0000000000001092

    Article  PubMed  Google Scholar 

Download references

Funding

The educational quality development fund at the Faculty of Health and Medical Sciences, University of Copenhagen, and the Toyota-Foundation, Denmark, provided funding for the study group to acquire the two Earsi simulators used in the study. SA has received research funding for an international postdoc from the Independent Research Fund Denmark (8026-00003B). The remaining authors have no other sources of funding or support to declare. Funding bodies were not involved in the conception, execution, interpretation, or publication of the study.

Author information

Affiliations

Authors

Contributions

MG, JM and SA were involved in the conception and design of the study. MG carried out data collection. JB, MF, JM and SA analysed the data. JB drafted the manuscript. MF, MG, JM and SA revised the manuscript critically for important intellectual content. All authors have approved the final version.

Corresponding author

Correspondence to Josefine Hastrup von Buchwald.

Ethics declarations

Conflicts of interest

None.

Ethics approval

The Regional Committee of Health Research Ethics of the Capital Region, Denmark, determined the study to be exempt (ref. 17006972).

Consent to participate

Informed consent was obtained from all participants in the study.

Consent for publication

Additional informed consent was obtained for publication of the article photograph.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

von Buchwald, J.H., Frendø, M., Guldager, M.J. et al. Content validity evidence for a simulation-based test of handheld otoscopy skills. Eur Arch Otorhinolaryngol 278, 2313–2320 (2021). https://doi.org/10.1007/s00405-020-06336-6

Download citation

Keywords

  • Handheld otoscopy
  • Simulation-based training
  • Technical skills training
  • Evidence-based medical education
  • Otology