The stochastically curtailed generalized likelihood ratio: A new termination criterion for variable-length computerized classification tests

Abstract

Computerized classification tests (CCTs) are used to classify examinees into categories in the context of professional certification testing. The term “variable-length” refers to CCTs that terminate (i.e., cease administering items to the examinee) when a classification can be made with a prespecified level of certainty. The sequential probability ratio test (SPRT) is a common criterion for terminating variable-length CCTs, but recent research has proposed more efficient methods. Specifically, the stochastically curtailed SPRT (SCSPRT) and the generalized likelihood ratio criterion (GLR) have been shown to classify examinees with accuracy similar to the SPRT while using fewer items. This article shows that the GLR criterion itself may be stochastically curtailed, resulting in a new termination criterion, the stochastically curtailed GLR (SCGLR). All four criteria—the SPRT, SCSPRT, GLR, and the new SCGLR—were compared using a simulation study. In this study, we examined the criteria in testing conditions that varied several CCT design features, including item bank characteristics, pass/fail threshold, and examinee ability distribution. In each condition, the termination criteria were evaluated according to their accuracy (proportion of examinees classified correctly), efficiency (test length), and loss (a single statistic combing both accuracy and efficiency). The simulation results showed that the SCGLR can yield increased efficiency without sacrificing accuracy, relative to the SPRT, SCSPRT, and GLR in a wide variety of CCT designs.

Appendixes

Appendix 1

We now review the computation of P(D _J  = D _j ′) originally given by Finkelman (2008).

Two methods of computation were proposed in that article, computing the probability exactly and approximating it by using the central limit theorem. Both methods rely on knowledge of the (j ′ + 1) … J remaining items that may potentially be administered to an examinee after stage j ′ if the test is not terminated before the maximum number of items J is reached. Finkelman (2008) notes that the remaining potential items can be known even if an item exposure control method is used provided the item selection method is non-adaptive.

When the number of remaining items (J − j ′) is relatively small, P(D _J = D _j ′) may be computed exactly. For example, suppose an examinee is at stage j ′ = 97 of a test with maximum number of items J = 100. Then, (J − j ′) = 3 items remain, and thus there are 2³ = 8 possible response patterns for those items. Given the parameters of those items and a θ value, two pieces of information may be determined: (1) the probability that each pattern occurs calculated using Eq. 1, and (2) the response patterns that lead to the interim decision matching the final decision—that is, for that patterns is it true that ${D}_J={D}_{j^{\hbox{'}}}$. The specific θ value used depends on D _j ′ as described by Finkelman (2008): θ ₊ is used if the interim classification decision at stage j ′ is nonmastery, and θ ₋ is used if the interim decision is mastery. Once this information is obtained, the probabilities for the response patterns for which D _J = D _j ′ are summed, resulting in an exact calculation of P(D _J = D _j ′).

P(D _J = D _j ′) may also be approximated using the following formulas, given by Finkelman (2008). For the case in which the interim decision is nonmastery, P(D _J = D _j ′) is given by

$$ P\left({D}_J={D}_{j\prime}\right)=P\left({D}_J=\mathrm{nonmastery}\right)\approx \Phi \left(\frac{ \log C-{E}_{\theta}\left(\left. \log {\lambda}_J\right|{\lambda}_{j\prime}\right)}{{\left\{ Va{r}_{\theta}\left(\left. \log {\lambda}_J\right|{\lambda}_{j\prime}\right)\right\}}^{\frac{1}{2}}}\right), $$

(A1)

where Φ is the cumulative distribution function of the standard normal distribution; E _θ (log λ _J |λ _j ′), the conditional expectation of log λ _J given λ _j ′, is given by

$$ {E}_{\theta}\left(\left. \log {\lambda}_J\right|{\lambda}_{j\prime}\right)= \log {\lambda}_{j\prime }+{\displaystyle {\sum}_{j=j\prime +1}^J}{E}_{\theta}\left( \log \frac{L\left({\theta}_{+};{X}_j\right)}{L\left({\theta}_{-};{X}_j\right)}\right); $$

(A2)

and the conditional variance is

$$ Va{r}_{\theta}\left(\left. \log {\lambda}_J\right|{\lambda}_{j\prime}\right)={\displaystyle {\sum}_{j=j\prime +1}^J} Va{r}_{\theta}\left( \log \frac{L\left({\theta}_{+};{X}_j\right)}{L\left({\theta}_{-};{X}_j\right)}\right). $$

(A3)

Then, for the case in which the interim decision is mastery, P(D _J = D _j ′) is given by

$$ P\left({D}_J=\mathrm{mastery}\right)=1-P\left({D}_J=\mathrm{nonmastery}\right). $$

Note that Eqs. A2 and A3 use the notation X _j , which represents the response to a single item. This is in contrast to the boldface notation used in Eq. 4, denoting a vector of responses.

Appendix 2

Table 6 48 simulation conditions, resulting from three levels of cut point (–1, 0, 1), two levels of bank shape (broad and peaked), two levels of δ (.10 and .20), two levels of γ (.80 and .99), and two levels of θ distribution [Normal (0,1) and Normal (0.25,1.25)]

Table 7 All simulation results yielded by the 48 conditions listed in Table 6