CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores
- 1.6k Downloads
Typically, only a minority of applicants to health professional training are invited to interview. However, pre-interview measures of cognitive skills predict for national licensure scores (Gauer et al. in Med Educ Online 21 2016) and subsequently licensure scores predict for performance in practice (Tamblyn et al. in JAMA 288(23): 3019–3026, 2002; Tamblyn et al. in JAMA 298(9):993–1001, 2007). Assessment of personal and professional characteristics, with the same psychometric rigour of measures of cognitive abilities, are needed upstream in the selection to health profession training programs. To fill that need, Computer-based Assessment for Sampling Personal characteristics (CASPer)—an on-line, video-based screening test—was created. In this paper, we examine the correlation between CASPer and Canadian national licensure examination outcomes in 109 doctors who took CASPer at the time of selection to medical school. Specifically, CASPer scores were correlated against performance on cognitive and ‘non-cognitive’ subsections of both the Medical Council of Canada Qualifying Examination (MCCQE) Parts I (end of medical school) and Part II (18 months into specialty training). Unlike most national licensure exams, MCCQE has specific subcomponents examining personal/professional qualities, providing a unique opportunity for comparison. The results demonstrated moderate predictive validity of CASPer to national licensure outcomes of personal/professional characteristics three to six years after admission to medical school. These types of disattenuated correlations (r = 0.3–0.5) are not otherwise predicted by traditional screening measures. These data support the ability of a computer-based strategy to screen applicants in a feasible, reliable test, which has now demonstrated predictive validity, lending evidence of its validation for medical school applicant selection.
KeywordsSelection Screening Situational-judgment test National licensing correlations Predictive validation Professionalism Non-academic qualities
The authors would like to acknowledge the generous contributions of the group at CAPER, including Steve Slade, for their actions matching data in a highly secure manner as well as the Medical Council of Canada (Margurite Roy) for the data collation. The authors wish to further acknowledge the support of McMaster faculty and students, who supported the implementation of CASPer, and particularly to Admissions Officer Wendy Edge, whose heroic efforts made implementation possible. Financial support for this project was received both from the Medical Council of Canada Medical Education Grant and the National Board of Medical Examiners (Stemmler Fund). The authors would like to acknowledge Ellen MacLellan for her assistance with the final preparation of the manuscript. KD, GN and HR are founders and KD and HR are board members of the company that now administers CASPer (Altus Assessments). All of the research noted above was completed prior to the establishment of Altus Assessments.
- Barlett, J. E., Kotrlik, J. W., & Higgins, C. C. (2001). Organizational research: Determining appropriate sample size in survey research. Information Technology, Learning, and Performance Journal, 19(1), 43.Google Scholar
- Dunleavy, D. M., & Whittaker, K. M. (2011). The evolving medical school admissions interview. AAMC Analysis in Brief, 11(7), 1–2.Google Scholar
- McManus, I. C., Dewberry, C., Nicholson, S., & Dowell, J. S. (2013a). The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools. BMC Medicine, 11(1), 1.CrossRefGoogle Scholar
- McManus, I. C., Dewberry, C., Nicholson, S., Dowell, J. S., Woolf, K., & Potts, H. W. (2013b). Construct-level predictive validity of educational attainment and intellectual apti tude tests in medical student selection: Meta-regression of six UK longitudinal stud ies. BMC Medicine, 11(1), 1.CrossRefGoogle Scholar