The UMP Exact Test and the Confidence Interval for Person Parameters in IRT Models

Abstract

In educational and psychological measurement when short test forms are used, the asymptotic normality of the maximum likelihood estimator of the person parameter of item response models does not hold. As a result, hypothesis tests or confidence intervals of the person parameter based on the normal distribution are likely to be problematic. Inferences based on the exact distribution, on the other hand, do not suffer from this limitation. However, the computation involved for the exact distribution approach is often prohibitively expensive. In this paper, we propose a general framework for constructing hypothesis tests and confidence intervals for IRT models within the exponential family based on exact distribution. In addition, an efficient branch and bound algorithm for calculating the exact p value is introduced. The type-I error rate and statistical power of the proposed exact test as well as the coverage rate and the lengths of the associated confidence interval are examined through a simulation. We also demonstrate its practical use by analyzing three real data sets.

Appendices

Appendix A

Under the 2PL model, the probability of a correct response for jth item from a subject is

$$\begin{aligned} P_j(X_j=1 | a_j,b_j,\theta ) = \frac{\exp [a_j(\theta -b_j)]}{1+\exp [a_j(\theta -b_j)]}, \end{aligned}$$

(13)

where $a_j$ is the item discrimination parameter, $b_j$ is the item difficulty parameter, and $\theta$ is the ability parameter for the subject. It follows that the likelihood of $\theta$ given a response pattern $\varvec{X} = \varvec{x}$ is

$$\begin{aligned} L\left( \theta |\varvec{x},\varvec{a},\varvec{b}\right)&= \prod _{j=1}^{J} P_j\left( X_j=1 | a_j,b_j,\theta \right) ^{x_j} P_j\left( X_j=0 | a_j,b_j,\theta \right) ^{1-x_j} \end{aligned}$$

(14)

$$\begin{aligned}&= \prod _{j=1}^{J} \left\{ \frac{\exp \left[ a_j\left( \theta -b_j\right) \right] }{1+\exp \left[ a_j\left( \theta -b_j\right) \right] }\right\} ^{x_j} \left\{ \frac{1}{1+\exp \left[ a_j\left( \theta -b_j\right) \right] }\right\} ^{1-x_j} \end{aligned}$$

(15)

$$\begin{aligned}&= \prod _{j=1}^{J} \frac{\left\{ \exp \left[ a_j\left( \theta -b_j\right) \right] \right\} ^{x_j}}{1+\exp \left[ a_j\left( \theta -b_j\right) \right] } \left\{ \frac{1}{1+\exp \left[ a_j\left( \theta -b_j\right) \right] }\right\} ^{x_j-x_j} \end{aligned}$$

(16)

$$\begin{aligned}&= \prod _{j=1}^{J} \frac{\left\{ \exp \left[ a_j\left( \theta -b_j\right) \right] \right\} ^{x_j}}{1+\exp \left[ a_j\left( \theta -b_j\right) \right] } \end{aligned}$$

(17)

$$\begin{aligned}&= \frac{\exp \left[ \sum _{j=1}^{J}x_j a_j\left( \theta -b_j\right) \right] }{\prod _{j=1}^{J} \left\{ 1+\exp \left[ a_j\left( \theta -b_j\right) \right] \right\} } \end{aligned}$$

(18)

$$\begin{aligned}&= \frac{\exp \left[ \theta \sum _{j=1}^{J}a_j x_j\right] }{\exp \left[ \sum _{j=1}^{J}a_j x_j b_j\right] \prod _{j=1}^{J} \left\{ 1+\exp \left[ a_j\left( \theta -b_j\right) \right] \right\} } \end{aligned}$$

(19)

$$\begin{aligned}&= \exp \left[ \theta \sum _{j=1}^{J}a_j x_j\right] \left\{ \exp \left[ \sum _{j=1}^{J}a_j x_j b_j\right] \right\} ^{-1} \left\{ \prod _{j=1}^{J} \left\{ 1+\exp \left[ a_j\left( \theta -b_j\right) \right] \right\} \right\} ^{-1}. \end{aligned}$$

(20)

Equation (20) is in the exponential form $L(\theta | \varvec{x}) = \exp [\eta (\theta )T(\varvec{x})]h(\varvec{x})g(\theta )$, where

$$\begin{aligned} \exp \left[ \eta \left( \theta \right) T\left( \varvec{x}\right) \right]&= \exp \left( \theta \sum _{j=1}^{n}a_j x_j\right) , \end{aligned}$$

(21)

$$\begin{aligned} h\left( \varvec{x}\right)&= \exp \left( \sum _{j=1}^{n}a_j x_j b_j\right) ^{-1}, \end{aligned}$$

(22)

and

$$\begin{aligned} g(\theta ) = \prod _{j=1}^{n}\{1-\exp [a_j(\theta -b_j)]\}^{-1}. \end{aligned}$$

(23)

Appendix B

In the food security data example, we are interested in testing the one-sided hypothesis: $H_0{:}\; \theta \le 1.93$ against $H_1{:}\; \theta \ge 1.93$. Using the exact test approach, the following response patterns are rejected at $\alpha =0.05$ level:

Table 2 18 Patterns that are rejected under the exact test.

Table 3 10 Patterns that are rejected under the asymptotic approach.