Optimization of answer keys for script concordance testing: should we exclude deviant panelists, deviant responses, or neither?
- First Online:
The Script Concordance Test (SCT) uses a panel-based, aggregate scoring method that aims to capture the variability of responses of experienced practitioners to particular clinical situations. The use of this type of scoring method is a key determinant of the tool’s discriminatory power, but deviant answers could potentially diminish the reliability of scores by introducing measurement error. (1) to investigate the effects on SCT psychometrics of excluding from the test’s scoring key either deviant panelists or deviant answers; (2) to propose a method for excluding either deviant panelists or deviant answers. Using an SCT in radiation oncology, we examined three methods for reducing panel response variability. One method (‘outliers’) entailed removing from the panel members with very low total scores. Two other methods (‘distance-from-mode’ and ‘judgment-by-experts’) excluded widely deviant responses to individual questions from the test’s scoring key. We compared the effects of these methods on score reliability, correlations between original and adjusted scores, and between-group effect sizes (panel-residents; panel-students; and residents-students). With a large panel (n = 45), optimization methods have no effect on reliability of scores, correlation and effect size. With a smaller panel (n = 15) no significant effect of optimization methods were observed on reliability and correlation, but significant variation on effect size was observed across samples. Measurement error resulting from deviant panelist responses on SCTs is negligible, provided the panel size is sufficiently large (>15). However, if removal of deviant answers is judged necessary, the distance-from-mode strategy is recommended.
KeywordsClinical Reasoning Script concordance test Optimization Reliability Panel
- Lambert, C., Gagnon, R., Nguyen, D., Charlin, B. (2009). The script concordance test in radiation oncology: Validation study of a new tool to assess clinical reasoning. Radiation Oncology, 9, 4–7. http://www.ro-journal.com/content/4/1/7.
- Lubarsky, S., Chalk, C., Kazitani, D., Gagnon, R., & Charlin, B. (2009). The script concordance test: A new tool assessing clinical judgment in neurology. Canadian Journal of Neurological Sciences, 36, 326–331.Google Scholar
- Lubarsky, S., Charlin, B., Cook, D.A., Chalk, C., & van der Vleuten, C. (2011). Script concordance method: A review of published validity evidence. Medical Education (in press).Google Scholar