Introducing a model for optimal design of sequential objective structured clinical examinations

Abstract

In a sequential OSCE which has been suggested to reduce testing costs, candidates take a short screening test and who fail the test, are asked to take the full OSCE. In order to introduce an effective and accurate sequential design, we developed a model for designing and evaluating screening OSCEs. Based on two datasets from a 10-station pre-internship OSCE and considering three factors, namely, the number of stations, the criteria for selecting the stations, and the cut-off score, several hypothetical tests were proposed. To investigate their accuracy, the positive predictive value (PPV), the pass rate, and the negative predictive value (NPV) were calculated. Also, a “desirable” composite outcome was defined as PPV = 100 %, pass rate ≥50 %, and NPV ≥25 %. Univariate and multiple logistic regression analyses were conducted to estimate the effects of independent factors on the occurrence of the desirable outcome. In half of the screening tests no false positive result was detected. Most of the screening OSCEs had acceptable levels of pass rate and NPV. Considering the desirable composite outcome 20 screening OSCEs could have successfully predicted the results of the corresponding full OSCE. The multiple regression analysis indicated significant contributions for the selection criteria (p values = 0.019) and the cut-off score (p values = 0.017). In order to have efficient screening OSCEs with the lowest probability of the error rate, careful selection of stations with high values of discrimination or item total correlation, and use of a relatively stringent cut-off score should be considered.