Surgical Education, Simulation, and Simulators—Updating the Concept of Validity
- 21 Downloads
Purpose of Review
Competency-based medical education (CBME) is rooted in the use of iterative assessments. We must ensure that the assessments used in CBME are valid, to make acceptable and accurate decisions regarding the competency of a trainee. Until recently, much of the educational and assessment literature in urology have used a now-outdated method of determining validity, based on theory and recommendations from over 50 years ago. We describe a contemporary approach to gathering construct validity evidence for the assessment of urologic trainees, for use in both clinical and simulation environments.
Five sources of evidence make up Messick’s contemporary framework of validity: test content, response process, internal structure, relationship to other variables, and consequences. These are all components of construct validation and concern the accuracy, quality, reproducibility, generalizability, and wider impact of the scores generated by an assessment, respectively.
When deciding the competency of a trainee, program directors and educators must have a clear understanding of how the validity is established and is determined in each assessment context. The contextual specificity of validity means that stakeholders must be prepared to defend the outcome of an assessment, particularly when making high-stake or summative decisions.
KeywordsSurgical education Simulation Validity Assessment Trainees Urology
Compliance with Ethical Standards
Conflict of Interest
Mitchell Goldenberg and Jason Y. Lee each declare no potential conflicts of interest.
Human and Animal Rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors.
Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance
- 3.• Aydin A, Raison N, Khan MS, Dasgupta P, Ahmed K. Simulation-based training and assessment in urological surgery. Nat Rev Urol. 2016;13(9):503–19. https://doi.org/10.1038/nrurol.2016.147. This article from 2015 summarizes simulation-based training interventions and assessment methods used across different urology surgical modalities. While thorough in its review of the literature at the time, it uses the now-outdated taxonomy of validity to describe and categorize the literature. This article provides the reader with an in-depth description of this outmoded validity language. CrossRefPubMedGoogle Scholar
- 5.education SDM, 2003. Validity: on the meaningful interpretation of assessment data. Wiley Online Library.Google Scholar
- 8.Association AER, Association AP, National Council on Measurement in Education, U.S JCOSFEAPT. Standards for Educational and Psychological Testing. Amer Educational Research Assn; 1999.Google Scholar
- 13.Association AER, Association AP, National Council on Measurement in Education, U.S JCOSFEAPT. Standards for Educational and Psychological Testing. 2014.Google Scholar
- 14.Messick S. Validity of psychological assessment. 1994.Google Scholar
- 17.Dagnaes-Hansen J, Mahmood O, Bube S, et al. Direct observation vs. video-based assessment in flexible cystoscopy. J Surg Educ. 2017; https://doi.org/10.1016/j.jsurg.2017.10.005.
- 23.•• Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014;19(2):233–50. https://doi.org/10.1007/s10459-013-9458-4. This article provides an extensive review, but more importantly description, of the simulation-based literature as framed by Messick’s framework. The various sources of validity evidence are described with examples from the literature, and the “data elements” that are encompassed by these domains are outlined as well. CrossRefPubMedGoogle Scholar
- 28.• Brydges R, Hatala R, Zendejas B, Erwin PJ, Cook DA. Linking simulation-based educational assessments and patient-related outcomes. Acad Med. 2015;90(2):246–56. https://doi.org/10.1097/ACM.0000000000000549. This article provides a good review of the literature relating simulation-based assessments with patient outcomes. Consequences evidence is a key element of the validity framework that has been vastly underexplored to date, and this article not only collates the available evidence but also clearly underscores the importance of this data in the design and implementation of competency-based assessments. CrossRefPubMedGoogle Scholar
- 29.Fecso AB, Szasz P, Kerezov G, Grantcharov TP. The effect of technical performance on patient outcomes in surgery: a systematic review. Ann Surg. August 2016; https://doi.org/10.1097/SLA.0000000000001959.