Optimal Bayesian Adaptive Design for Test-Item Calibration

Abstract

An optimal adaptive design for test-item calibration based on Bayesian optimality criteria is presented. The design adapts the choice of field-test items to the examinees taking an operational adaptive test using both the information in the posterior distributions of their ability parameters and the current posterior distributions of the field-test parameters. Different criteria of optimality based on the two types of posterior distributions are possible. The design can be implemented using an MCMC scheme with alternating stages of sampling from the posterior distributions of the test takers’ ability parameters and the parameters of the field-test items while reusing samples from earlier posterior distributions of the other parameters. Results from a simulation study demonstrated the feasibility of the proposed MCMC implementation for operational item calibration. A comparison of performances for different optimality criteria showed faster calibration of substantial numbers of items for the criterion of D-optimality relative to A-optimality, a special case of c-optimality, and random assignment of items to the test takers.

Appendices

Appendix A. Implementations of the MCMC Algorithm

The MCMC algorithms used for the calibration design are variations of the usual MH within Gibbs algorithm with blocks of item and examinee parameters and symmetric proposal densities for the 3PL model. The general structure of the algorithm has been documented extensively (e.g., Fox 2010, Chapter 3; Johnson & Albert, 1999, Section 2.5; Patz & Junker, 1999a, 1999b). The following two versions of the algorithm are used.

1.1 A.1 Posterior of Ability Parameter

The first version is for the draws from the posterior distribution $g(\theta_{j}\mid \mathbf{u}_{i_{k-1}})$ of ability parameter θ for test taker j in (2) upon answers to the items l=1,…,k−1 in the adaptive test. The posterior distributions of their parameters, $g(\boldsymbol {\eta }_{i_{l}})$, are assumed to be available in the system in the form of vectors with random draws $\boldsymbol {\eta }_{i_{l}}^{(t)}= (\boldsymbol {\eta }_{i_{l}}^{(1)},\ldots\boldsymbol {\eta }_{i_{l}}^{(T)})$.

The version can be summarized as iterations r=1,…,R each consisting of the following two steps:

• 1. The rth draw from the posterior distribution of θ for test taker j is obtained by

  • (a) Drawing a candidate value $\theta _{j}^{(c)}$ for $\theta _{j}^{(r)}$ from the proposal density $q(\theta _{j}\mid \theta _{j}^{(r-1)})$;

  • (b) Accepting $\theta _{j}^{(r)}=\theta _{j}^{(c)}$ with probability

    $$ \min \Biggl\{ \frac{g(\theta _{j}^{(c)})\prod_{l=1}^{k-1}p(\theta _{j}^{ ( c ) };\boldsymbol {\eta }_{i_{l}}^{(r-1)})^{u_{i_{l}}}[1-p(\theta _{j}^{(c)};\boldsymbol {\eta }_{i_{l}}^{(r-1)})]^{1-u_{i_{l}}}}{g(\theta _{j}^{(r-1)})\prod_{l=1}^{k-1}p(\theta _{j}^{ ( r-1 ) };\boldsymbol {\eta }_{i_{l}}^{(r-1)})^{u_{i_{l}}}[1-p(\theta _{j}^{(r-1)}; \boldsymbol {\eta }_{i_{l}}^{(r-1)})]^{1-u_{i_{l}}}},1 \Biggr\} $$

    (A.1)

    Otherwise, $\theta _{j}^{(r)}=\theta _{j}^{(r-1)}$.

• 2. The rth draws from the posterior distributions of the operational parameters $\eta _{i_{l}}$, l=1,…,k−1, are randomly sampled from the vectors $\boldsymbol {\eta }_{i_{l}}^{(t)}$ present in the system.

Upon stationarity, the draws from the posterior distribution of θ for test taker j in the first steps collected in a vector $\boldsymbol {\theta }_{j}^{(s)}= (\theta _{j}^{(1)},\ldots,\theta _{j}^{(S)})$.

1.2 A.2 Update of Posterior of Field-Test Parameters

The second version is for the updates of the posterior distributions of the parameters of field-test item f after batch b _f =1,2,… of test takers. The current posterior distributions of the ability parameters θ _j for the test takers j∈b _f are available in the form of estimates of their densities g(θ _j ) derived from the vectors with the draws $\boldsymbol {\theta }_{j}^{(s)}=(\theta _{j}^{(1)},\ldots,\theta _{j}^{(S)})$. Likewise, the current posterior distributions of the field-test parameter η _f , f=1,…,F, are available in the form of estimates of their densities $g^{(b-1)}(\boldsymbol {\eta }_{f}\mid \mathbf{u}_{f_{j}})$ derived from the vectors with random draws $\boldsymbol {\eta }_{f}^{(b-1,t)}=(\boldsymbol {\eta }_{f}^{(b-1,1)},\ldots, \boldsymbol {\eta }_{f}^{(b-1,T)})$. See the main text for the derivation of these density estimates.

The second version can be summarized as iterations r=1,…,R each consisting of the following two steps:

• 1. For each j∈b, the rth draw from the posterior distribution of θ _j is obtained by

  • (a) Drawing a candidate value $\theta _{j}^{(c)}$ for $\theta _{j}^{(r)}$ from the proposal density $q(\theta _{j}\mid \theta _{j}^{(r-1)})$;

  • (b) Accepting $\theta _{j}^{(r)}=\theta _{j}^{(c)}$ with probability

    $$ \min \Biggl\{ \frac{g(\theta _{j}^{(c)})\prod_{f=1}^{F}p(\theta_{j}^{ ( c ) }; \boldsymbol {\eta }_{f}^{(r-1)})^{u_{f_{j}}}[1-p(\theta_{j}^{(c)}; \boldsymbol {\eta }_{f}^{(r-1)})]^{1-u_{f_{j}}}}{g(\theta_{j}^{(r-1)}) \prod_{f=1}^{F}p(\theta _{j}^{ ( r-1 ) };\boldsymbol {\eta }_{f}^{(r-1)})^{u_{f_{j}}}[1-p(\theta _{j}^{(r-1)}; \boldsymbol {\eta}_{f}^{(r-1)})]^{1-u_{f_{j}}}},1 \Biggr\} $$

    (A.2)

    Otherwise, $\theta _{j}^{(r)}=\theta _{j}^{(r-1)}$.

• 2. For each of the field-test items f administered to a test taker j∈b _f , the rth draw from the posterior distribution of η _f is obtained by

  • (a) Drawing a candidate value $\boldsymbol {\eta }_{f}^{(c)}$ for $\boldsymbol {\eta}_{f}^{(r)}$ from a proposal density $q(\boldsymbol {\eta }_{f}\mid \boldsymbol {\eta }_{f}^{(r-1)})$;

  • (b) Accepting $\boldsymbol {\eta }_{f}^{(r)}=\boldsymbol {\eta }_{f}^{(c)}$ with probability

    $$ \min \Biggl\{ \frac{g^{(b-1)}(\boldsymbol {\eta }_{f}^{(c)}) \prod_{j=1}^{n_{b_{f}}} \{ p(\theta _{j}^{(r)};\boldsymbol {\eta }_{f}^{(c)})^{u_{f_{j}}} [1-p(\theta _{j}^{(r)};\boldsymbol {\eta }_{f}^{(c)})]^{1-u_{f_{j}}} \} }{g^{(c)}(\boldsymbol {\eta }_{f}^{(r-1)}) \prod_{j=1}^{n_{b_{f}}} \{ p(\theta _{j}^{ (r ) }; \boldsymbol {\eta }_{f}^{(r-1)})^{u_{f_{j}}}[1-p(\theta _{j}^{(r)}; \boldsymbol {\eta }_{f}^{(r-1)})]^{1-u_{f_{j}}} \} },1 \Biggr\} $$

    (A.3)

    Otherwise, $\boldsymbol {\eta }_{f}^{(r)}=\boldsymbol {\eta }_{f}^{(r-1)}$.

Upon stationarity, the draws $\boldsymbol {\eta }_{f}^{(b,t)}=(\boldsymbol {\eta }_{f}^{(b,1)},\ldots, \boldsymbol {\eta }_{f}^{(b,T)})$ from the posterior update of η _f for batch b during the second steps are saved as updates of the vectors $\boldsymbol {\eta }_{f}^{(b-1,t)}$.

Appendix B. Information Matrices

2.1 B.1 Observed Information Matrix

For the 3PL model, the observed information matrix $J_{u_{f_{j}}}(\boldsymbol {\eta }_{f};\theta _{j})$ in (17) has entries

$$\begin{aligned} J_{u_{f}}(a_{f},a_{f};\theta ) =& -(1-p_{f}) (p_{f}-c_{f}) \bigl(u_{f}c_{f}-p_{f}^{2} \bigr) \biggl[ \frac{\theta -b_{f}}{(1-c_{f})p_{f}} \biggr] ^{2}; \end{aligned}$$

(B.1)

$$\begin{aligned} J_{u_{f}}(b_{f},b_{f};\theta ) =& -(1-p_{f}) (p_{f}-c_{f}) \bigl(u_{f}c_{f}-p_{f}^{2} \bigr) \biggl[ \frac{a_{f}}{(1-c_{f})p_{f}} \biggr] ^{2}; \end{aligned}$$

(B.2)

$$\begin{aligned} J_{u_{f}}(c_{f},c_{f};\theta ) =& \frac{u_{f}-2u_{f}p_{f}+p_{f}^{2}}{(1-c_{f})^{2}p_{f}^{2}}; \end{aligned}$$

(B.3)

$$\begin{aligned} J_{u_{f}}(a_{f},b_{f};\theta ) =& \frac{(p_{f}-c_{f})}{(1-c_{f})p_{f}^{2}} \biggl[ p_{f}(u_{f}-p_{f})+a_{f} \bigl(u_{f}c_{f}-p_{f}^{2}\bigr) \frac{(\theta -b_{f})(1-p_{f}) }{1-c_{f}} \biggr] ; \end{aligned}$$

(B.4)

$$\begin{aligned} J_{u_{f}}(a_{f},c_{f};\theta ) =& \frac{u_{f}(\theta -b_{f})(1-p_{f})(p_{f}-c_{f})}{(1-c_{f})^{2}p_{f}^{2}}; \end{aligned}$$

(B.5)

$$\begin{aligned} J_{u_{f}}(b_{f},c_{f};\theta ) =& \frac{-u_{f}a_{f}(1-p_{f})(p_{f}-c_{f})}{(1-c_{f})^{2}p_{f}^{2}}, \end{aligned}$$

(B.6)

where p _f is the response probability on field-test item f in (1).

2.2 B.2 Expected Information Matrix

The expected matrix $I_{U_{f}}(\boldsymbol {\eta }_{f};\theta _{j})$ in (9) is readily available in the literature (e.g., Lord 1980, Section 12.1). Using the notation in this paper, it is obtained taking the expectations of (B.1)–(B.6) over the response distribution, as

$$\begin{aligned} I_{U_{f}}(a_{f},a_{f};\theta ) =& \frac{(1-p_{f})(p_{f}-c_{f})^{2}(\theta-b_{f})^{2}}{p_{f}(1-c_{f})^{2}}; \end{aligned}$$

(B.7)

$$\begin{aligned} I_{U_{f}}(b_{f},b_{f};\theta ) =& \frac{a_{f}^{2}(1-p_{f})(p_{f}-c_{f})^{2}}{p_{f}(1-c_{f})^{2}}; \end{aligned}$$

(B.8)

$$\begin{aligned} I_{U_{f}}(c_{f},c_{f};\theta ) =& \frac{(1-p_{f})}{p_{f}(1-c_{f})^{2}}; \end{aligned}$$

(B.9)

$$\begin{aligned} I_{U_{f}}(a_{f},b_{f};\theta ) =& \frac{-a_{f}(1-p_{f})(p_{f}-c_{f})^{2}(\theta -b_{f})}{p_{f}(1-c_{f})^{2}}; \end{aligned}$$

(B.10)

$$\begin{aligned} I_{U_{f}}(a_{f},c_{f};\theta ) =& \frac{(1-p_{f})(p_{f}-c_{f})(\theta -b_{f})}{p_{f}(1-c_{f})^{2}}; \end{aligned}$$

(B.11)

$$\begin{aligned} I_{U_{f}}(b_{f},c_{f};\theta ) =& \frac{-a_{f}(1-p_{f})(p_{f}-c_{f})}{p_{f}(1-c_{f})^{2}}. \end{aligned}$$

(B.12)

2.3 B.3 Observed Information Matrix for Transformed Parameters

For a multivariate normal proposal distribution based on the transformations $a_{f}^{\ast }=\ln a_{f}$ and $c_{f}^{\ast }=\operatorname {logit}c_{f}$ in (5), the entries of the observed information matrix in (B.1)–(B.6) take the following form:

$$\begin{aligned} J_{u_{f}}^{\ast }(a_{f},a_{f};\theta ) =& \frac{-a_{f}(\theta -b_{f})(u_{f}-p_{f})(p_{f}-c_{f})}{p_{f}(1-c_{f})} \\ &{}-a_{f}^{2}(1-p_{f}) (p_{f}-c_{f}) \bigl(u_{f}c_{f}-p_{f}^{2}\bigr) \biggl[ \frac{\theta-b_{f}}{p_{f}(1-c_{f})} \biggr] ^{2}; \end{aligned}$$

(B.13)

$$\begin{aligned} J_{u_{f}}^{\ast }(b_{f},b_{f};\theta) =& -(1-p_{f}) (p_{f}-c_{f}) \bigl(u_{f}c_{f}-p_{f}^{2}\bigr) \biggl[ \frac{a_{f}}{p_{f}(1-c_{f})} \biggr] ^{2}; \end{aligned}$$

(B.14)

$$\begin{aligned} J_{u_{f}}^{\ast }(c_{f},c_{f};\theta ) =& \frac{-(c_{f}-3c_{f}^{2}+2c_{f}^{3})(u_{f}-p_{f})}{p_{f}(1-c_{f})} +\frac{c_{f}^{2}(u_{f}-2u_{f}p_{f}+p_{f}^{2})}{p_{f}^{2}}; \end{aligned}$$

(B.15)

$$\begin{aligned} J_{u_{f}}^{\ast }(a_{f},b_{f};\theta ) =& \frac{a_{f}(p_{f}-c_{f})}{(1-c_{f})p_{f}^{2}} \\ &{}\times \biggl[ p_{f}(u_{f}-p_{f})+a_{f} \bigl(u_{f}c_{f}-p_{f}^{2}\bigr) \frac{(\theta -b_{f})(1-p_{f})}{1-c_{f}} \biggr] ; \end{aligned}$$

(B.16)

$$\begin{aligned} J_{u_{f}}^{\ast }(a_{f},c_{f};\theta ) =& \frac{u_{f}a_{f}c_{f}(\theta -b_{f})(1-p_{f})(p_{f}-c_{f})}{(1-c_{f})p_{f}^{2}}; \end{aligned}$$

(B.17)

$$\begin{aligned} J_{u_{f}}^{\ast }(b_{f},c_{f};\theta ) =& \frac{-u_{f}a_{f}c_{f}(1-p_{f})(p_{f}-c_{f})}{(1-c_{f})p_{f}^{2}}. \end{aligned}$$

(B.18)

Observe that, before entering the acceptance criterion in (A.2), the draws of $a_{f}^{\ast }$ and $c_{f}^{\ast }$ from the proposal distribution with this version of covariance matrix have to transformed back to their original scales as $a_{f}=\exp (a_{f}^{\ast })$ and $c_{f}=[1+\exp (-c_{f}^{\ast})]^{-1}$.