Applying Kane’s validity framework to a simulation based assessment of clinical competence
- 621 Downloads
Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane’s validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating “Scoring” included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating “Generalizability” included: (a) a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating “Extrapolation” included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating “Implications” included low error rates in practice. Interpreting our findings according to Kane’s framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane’s validity framework to evaluate, and iteratively develop and refine assessment strategies.
KeywordsAssessment Competence OSCE Paramedic Simulation Validation Validity
The authors would like to thank the Ontario Base Hospital Group for their support in completing this study.
- Cook, D. A., & Hatala, R. (2016). Validation of educational assessments: A primer for simulation and beyond. Advances in Health Sciences Education Theory and Practice, 1(1), 31.Google Scholar
- Cronbach, L. J. (1989). Construct validation after thirty years. Intelligence Measurement Theory and Public Policy, 3, 147–171.Google Scholar
- Frank, J., Snell, L., & Sherbino, J. (2014). Draft Can-MEDS 2015 physician competency framework-series III. Ottawa, Ontario: The Royal College of Physicians and Surgeons of Canada.Google Scholar
- Hatala, R., Cook, D. A., Brydges, R., & Hawkins, R. (2015). Constructing a validity argument for the objective structured assessment of technical skills (OSATS): A systematic review of validity evidence. Advances in Health Sciences Education Theory and Practice, 20(5), 1149–1175.CrossRefGoogle Scholar
- Kane, M. T. (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 20(10), 5–17.Google Scholar
- Kane, M. T. (2013b). Validity. In B. L. Brennan (Ed.), Educational measurement. Westport, CT: Praeger Publishers.Google Scholar
- Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement. McMillan: Old Tappan, NJ.Google Scholar
- Ministry of Health and Long-Term Care, Emergency Health Services Branch. (2007). Basic life support patient care standards version 2.0. Toronto, Ontario: Publications Ontario.Google Scholar
- Ministry of Health and Long-Term Care, Emergency Health Services Branch. (2015). Advanced life support patient care standards version 3.2. Toronto, Ontario: Publications Ontario.Google Scholar
- Paramedic Association of Canada. (2016). National occupational competency profile 2016. Retrieved February 22, 2017, from http://paramedic.ca/site/nocp?nav=02.
- Sunnybrook Centre for Prehospital Medicine. (2016). Regional Base Hospital 2015–2016 annual report. Toronto: Ontario.Google Scholar