Optimal and Most Exact Confidence Intervals for Person Parameters in Item Response Theory Models

Abstract

The common way to calculate confidence intervals for item response theory models is to assume that the standardized maximum likelihood estimator for the person parameter θ is normally distributed. However, this approximation is often inadequate for short and medium test lengths. As a result, the coverage probabilities fall below the given level of significance in many cases; and, therefore, the corresponding intervals are no longer confidence intervals in terms of the actual definition. In the present work, confidence intervals are defined more precisely by utilizing the relationship between confidence intervals and hypothesis testing. Two approaches to confidence interval construction are explored that are optimal with respect to criteria of smallness and consistency with the standard approach.

Appendix: Mathematical Proofs

The following proves Proposition 3.1.

Proof

Note that the prior g is derived by a simple application of integration by substitution. To show that τ ⁻¹ I is WAMA we first observe that for any confidence interval J and any $\tilde{\rho}$ with $\tilde{\theta}= \tau(\tilde{\rho})$,

$$\int_{\varTheta}\mathrm{P}_\theta(\tilde{\theta}\in J) f( \theta) \mathrm{d}\theta= \int_{U}\mathrm{P}_\rho \bigl(\tilde{\rho}\in\tau^{-1}J\bigr) g(\rho) \mathrm{d}\rho. $$

If J is now any confidence interval on U, then

$$\int_{U}\mathrm{P}_\rho\bigl(\tilde{\rho}\in \tau^{-1}I\bigr) g(\rho) \mathrm{d}\rho\leq\int_{U} \mathrm{P}_\rho(\tilde{\rho}\in J) g(\rho) \mathrm{d}\rho; $$

otherwise J would induce, with obvious notation, an interval τJ on ℝ with

$$\int_{\varTheta}\mathrm{P}_\theta(\tilde{\theta}\in I) f( \theta) \mathrm{d}\theta> \int_{\varTheta}\mathrm{P}_\theta( \tilde{\theta}\in\tau J) f(\theta) \mathrm{d}\theta, $$

in contradiction to the fact that I is WAMA on Θ. □

The following proves Proposition 4.4.

Proof

Let k,k′ be test results with the same total score R, and let β ₁,…,β _N be an enumeration of all item difficulty parameters of the test and, without loss of generality, let R≤N be such that β ₁,…,β _R are the item difficulty parameters of the items which have to be correctly solved in order to obtain k. Let $\beta_{1}', \ldots,\beta_{R}'$ denote the parameters of the items which have to be solved to obtain k′. For r=1,…,R let $x_{r}:=\beta_{r}-\beta_{r}'$. Furthermore, let $Q(\theta):=\prod_{i=1}^{N}(1+\exp(\theta-\beta_{i}))$. Then

 □

The following proves Proposition 4.3.

Proof

It suffices to show that for every possible test result k

$$ I_e(k)=I_n(k) $$

(A.1)

holds. For a given k and arbitrary θ we have

here the second equivalence holds because of the normal assumption. Therefore (A.1) is shown. □

The following proves Theorem 4.1.

Proof

Let $\tilde{\theta}\in\varTheta$ be arbitrary. We need to show

$$\int_\theta\mathrm{P}_\theta(\tilde{\theta}\in I_h) f(\theta) \mathrm{d}\theta= \min_{ \{I: I \mathrm{is\ an}\ (1- \alpha)\hbox{-}\mathrm{confidence}\ \mathrm{interval} \}}\int _\theta\mathrm {P}_\theta(\tilde{\theta}\in I) f(\theta) \mathrm{d}\theta. $$

Rearranging terms yields

$$\int_\theta\mathrm{P}_\theta(\tilde{\theta}\in I_h) f(\theta) \mathrm{d}\theta= \int_\theta\sum _{k \in A_{I_h,\tilde{\theta}}} \mathrm{P}_\theta(k) f(\theta) \mathrm{d}\theta= \sum_{k \in A_{I_h,\tilde{\theta}}}\int_\theta \mathrm{P}_\theta(k) f(\theta) \mathrm{d}\theta. $$

Let I be an alternative confidence interval for level 1−α with corresponding sets A _I,θ :={k:θ∈I(k)} for all θ. Define

We have to show

$$ \sum_{k\in C} \int_\theta \mathrm{P}_\theta(k)f(\theta)\mathrm {d}\theta\leq\sum _{k \in B} \int_\theta\mathrm{P}_\theta (k)f(\theta)\mathrm{d}\theta. $$

(A.2)

Because of the definition of I _h ,

$$ \forall k \in B\colon\quad \mathrm{c}_{\tilde{\theta}}\mathrm{P}_{\tilde{\theta}}(k)\leq\int _{\theta}\mathrm{P}_{\theta}(k)f(\theta) \mathrm {d}\theta $$

and

$$ \forall k \in C\colon\quad \int_{\theta}\mathrm{P}_{\theta}(k)f( \theta) d(\theta)\leq\mathrm{c}_{\tilde{\theta}}\mathrm{P}_{\tilde{\theta}}(k) $$

hold. These equations can be summed up over B and C, respectively. Therefore we obtain

$$ \mathrm{c}_{\tilde{\theta}} \sum_{k \in B} \mathrm{P}_{\tilde{\theta}}(k)\leq\sum_{k\in B}\int _{\theta}\mathrm{P}_{\theta}(k)f(\theta ) \mathrm{d}\theta $$

(A.3)

and

$$ \sum_{k \in C}\int_{\theta} \mathrm{P}_{\theta}(k)f(\theta) d(\theta)\leq\mathrm{c}_{\tilde{\theta}}\sum _{k \in C} \mathrm{P}_{\tilde{\theta}}(k). $$

(A.4)

Since I is a confidence interval for the level 1−α and I _h meets the level exactly, it is known that

$$\sum_{k\in C}\mathrm{P}_{\tilde{\theta}}(k)\leq\sum _{k\in B}\mathrm {P}_{\tilde{\theta}}(k). $$

From (A.3) and (A.4), (A.2) follows. □