Abstract
Objective measurement of simulation performance requires a validated and reliable tool. However, no published Italian language assessment tool is available. Translation of a published English language tool, the Ottawa Crisis Resource Management Global Rating Scale (GRS), may lead to a validated and reliable tool. After developing an Italian language translation of the English language tool, the study measured the reliability of the new tool by comparison with the English language tool used independently in the same simulation scenarios. In addition, the validity of the Italian language tool was measured by comparison to a skills score also applied independently. The correlation coefficient between the Italian language overall GRS and the English language overall GRS was 0.82 (adjusted 95 % confidence interval: 0.62–0.92). The correlation coefficient between the Italian language overall GRS and the skill score was 0.85 (adjusted 95 % confidence interval 0.68–0.94). This study demonstrated that the Italian language GRS has acceptable reliability when compared with the English language tool, suggesting that it can be used reliably to evaluate the performance during simulated emergencies. The study also suggests that the tool has acceptable validity for assessing the simulation performance. The study suggests that the Italian language GRS translation has reasonable reliability when compared with the English language GRS and reasonable validity when compared with the assessment of the skills scores. Data suggest that the instrument is adequately reliable for informal and formative type of examinations, but may require further confirmation before use for high-stake examinations such as licensing.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ericsson KA, Krampe RT, Tesch-Römer C (1993) The role of deliberate practice in the acquisition of expert performance. Psychol Rev 100:363–406
Gibbs G (1988) Learning by doing: a guide to teaching and learning methods. FEU
Grant J (1992) Training senior house officers by service-based learning. Joint centre for education in medicine, London
Kolb DA (2014) Experiential learning: experience as the source of learning and development. Pearson Education
Levine AI, Schwartz AD, Bryson EO, Demaria S (2012) Role of simulation in US physician licensure and certification. Mt Sinai J Med 79:140–153
Buyske J (2010) The role of simulation in certification. Surg Clin North Am 90:619–621
Gordon JA, Tancredi DN, Binder WD, Wilkerson WM, Shaffer DW (2003) Assessment of a clinical performance evaluation tool for use in a simulator-based testing environment: a pilot study. Acad Med 78:S45–S47
Epstein RM (2007) Medical education—assessment in medical education. N Engl J Med 356:387–396
Downing SM (2004) Reliability: on the reproducibility of assessment data. Med Educ 38:1006–1012
Downing SM (2003) Validity: on the meaningful interpretation of assessment data. Med Educ 37:830–837
Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J (2006) A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: the University of Ottawa Critical Care Medicine, high-fidelity simulation, and crisis resource management I study. Crit Care Med 34:2167–2174
Okuda Y, Godwin SA, Jacobson L, Wang E, Weingart S (2014) SimWars. J Emerg Med 47:586–593
Devore J (2011) Probability and statistics for engineering and the sciences. Thomson Brooks/Cole, USA
Litwin MS (1995) How to measure survey reliability and validity. SAGE Publications, USA
Hsu J (1996) Multiple comparisons: theory and methods. CRC Press, USA
Peng RD (2011) Reproducible research in computational science. Science 334:1226–1227
Franc JM (2016) Reveal project. http://www.medstatstudio.com/studies/project.php?pid=15. Accessed 18 Mar 2016
Montgomery DC (2009) Design and analysis of experiments. Wiley, USA
Flin R, Yule S, Paterson-Brown S, Maran N, Rowley D, Youngson G (2007) Teaching surgeons about non-technical skills. Surgeon 5:86–89
Powers KA, Rehrig ST, Irias N, Albano HA, Malinow A, Jones SB, Moorman DW, Pawlowski JB, Jones DB (2008) Simulated laparoscopic operating room crisis: an approach to enhance the surgical team performance. Surg Endosc 22:885–900
McCulloch P, Mishra A, Handa A, Dale T, Hirst G, Catchpole K (2009) The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Car 18:109–115
Riem N, Boet S, Bould MD, Tavares W, Naik VN (2012) Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 109:723–728
Brunckhorst O, Shahid S, Aydin A, Khan S, Mcllhenny C, Brewin J, Sahai A, Bello F, Kneebone R, Shamim Khan M, Dasgupta P, Ahmed K (2015) The relationship between technical and nontechnical skills within a simulation-based ureteroscopy training environment. J Surg Educ 72(5):1039–1044
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that they have no conflict of interest.
Statement of human and animal rights
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institution an/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Franc, J.M., Verde, M., Gallardo, A.R. et al. An Italian version of the Ottawa Crisis Resource Management Global Rating Scale: a reliable and valid tool for assessment of simulation performance. Intern Emerg Med 12, 651–656 (2017). https://doi.org/10.1007/s11739-016-1486-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11739-016-1486-7