Skip to main content
Log in

Efficient Standard Error Formulas of Ability Estimators with Dichotomous Item Response Models

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

This paper focuses on the computation of asymptotic standard errors (ASE) of ability estimators with dichotomous item response models. A general framework is considered, and ability estimators are defined from a very restricted set of assumptions and formulas. This approach encompasses most standard methods such as maximum likelihood, weighted likelihood, maximum a posteriori, and robust estimators. A general formula for the ASE is derived from the theory of M-estimation. Well-known results are found back as particular cases for the maximum and robust estimators, while new ASE proposals for the weighted likelihood and maximum a posteriori estimators are presented. These new formulas are compared to traditional ones by means of a simulation study under Rasch modeling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques. New York: Marcel Dekker.

  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (chapters) (pp. 17–20). Reading, MA: Addison-Wesley.

    Google Scholar 

  • Birnbaum, A. (1969). Statistical theory for logistic mental test models with a prior distribution of ability. Journal of Mathematical Psychology, 6, 258–276.

    Article  Google Scholar 

  • Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a micro computer environment. Applied Psychological Measurement, 6, 431–444.

  • Carroll, R. J., & Pederson, S. (1993). On robustness in the logistic regression model. Journal of the Royal Statistical Society: Series B, 55, 693–706.

  • Doebler, A. (2012). The problem of bias in person parameter estimation in adaptive testing. Applied Psychological Measurement, 36, 255–270. doi:10.1177/0146621612443304.

    Article  Google Scholar 

  • Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. New York: Erlbaum.

  • Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer.

  • Huber, P. J. (1964). Robust estimation of a location parameter. Annals of Mathematical Statistics, 35, 73–101. doi:10.1214/aoms/1177703732.

    Article  Google Scholar 

  • Huber, P.J. (1967). The behavior of maximum likelihood estimates under non-standard conditions. In Proceeding of the 5th Berkeley Symposium, (vol. 1, pp. 221–233).

  • Huber, P. J. (1981). Robust statistics. New York: Wiley.

    Book  Google Scholar 

  • Koralov, L., & Sinai, Y. G. (2007). Theory of probability and random processes. New York: Springer.

  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Lord, F. M. (1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48, 233–245. doi:10.1007/BF02294018.

    Article  Google Scholar 

  • Lord, F. M. (1986). Maximum likelihood and Bayesian parameter estimation in item response theory. Journal of Educational Measurement, 23, 157–162. doi:10.1111/j.1745-3984.1986.tb00241.x.

    Article  Google Scholar 

  • Magis, D. (2014a). On the asymptotic standard error of a class of robust estimators of ability in dichotomous item response models. British Journal of Mathematical and Statistical Psychology, 67, 430–450. doi:10.1111/bmsp.12027.

    Article  PubMed  Google Scholar 

  • Magis, D. (2014b). Accuracy of asymptotic standard errors of the maximum and weighted likelihood estimators of proficiency levels with short tests. Applied Psychological Measurement, 38, 105–121. doi:10.1177/0146621613496890.

    Article  Google Scholar 

  • Magis, D., & Raîche, G. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48, 1–31.

  • Mislevy, R. J. (1986). Bayes modal estimation in item response theory. Psychometrika, 51, 177–195. doi:10.1007/BF02293979.

    Article  Google Scholar 

  • Mislevy, R. J., & Bock, R. D. (1982). Biweight estimates of latent ability. Educational and Psychological Measurement, 42, 725–737. doi:10.1177/001316448204200302.

  • Mosteller, F., & Tukey, J. (1977). Exploratory data analysis and regression. Reading, MA: Addison-Wesley.

  • Nydick, S.W. (2013). catIrt: An R package for simulating IRT-based computerized adaptive tests. R package version 0.4-1.

  • Ogasawara, H. (2013a). Asymptotic properties of the Bayes and pseudo Bayes estimators of ability in item response theory. Journal of Multivariate Analysis, 114, 359–377. doi:10.1016/j.jmva.2012.08.013.

    Article  Google Scholar 

  • Ogasawara, H. (2013b). Asymptotic cumulants of the ability estimators using fallible item parameters. Journal of Multivariate Analysis, 119, 144–162. doi:10.1016/j.jmva.2013.04.008.

  • Partchev, I. (2012). irtoys: Simple interface to the estimation and plotting of IRT models. R package version 0.1.6.

  • Patton, J. M., Cheng, Y., Yuan, K.-H., & Diao, Q. (2013). The influence of item calibration error on variable-length computerized adaptive testing. Applied Psychological Measurement, 37, 24–40. doi:10.1177/0146621612461727.

  • R Core Team (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

  • Rao, M. M. (1984). Probability theory with applications. New York: Academic Press.

    Google Scholar 

  • Reif, M. (2014). PP: Estimation of person parameters for the 1, 2, 3, 4-PL model and the GPCM. R package version 0.5.3.

  • Schuster, C., & Yuan, K.-H. (2011). Robust estimation of latent ability in item response models. Journal of Educational and Behavioral Statistics, 36, 720–735. doi:10.3102/1076998610396890.

    Article  Google Scholar 

  • Sijisma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.

  • Stefanski, L. A., & Boos, D. D. (2002). The calculus of M-estimation. The American Statistician, 56, 29–38. doi:10.1198/000313002753631330.

  • Wainer, H. (2000). Computerized adaptive testing: A primer (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Wainer, H., & Wright, B. D. (1980). Robust estimation of ability in the Rasch model. Psychometrika, 45, 373–391. doi:10.1007/BF02293910.

  • Warm, T. A. (1989). Weighted likelihood estimation of ability in item response models. Psychometrika, 54, 427–450. doi:10.1007/BF02294627.

    Article  Google Scholar 

  • Warm, T. A. (2007). Warm (Maximum) likelihood estimates of Rasch measures. Rasch Measurement Transactions, 21, 1094.

    Google Scholar 

  • Wu, M. L., Adams, R. J., & Wilson, M. R. (1997). ConQuest: Multi-aspect test software [Computer program]. Camberwell, Australia: Australian Council for Educational Research.

  • Yuan, K.-H., & Jennrich, R. I. (1998). Asymptotics of estimating equations under natural conditions. Journal of Multivariate Analysis, 65, 245–260. doi:10.1006/jmva.1997.1731.

  • Zeileis, A. (2006). Object-oriented computation of sandwich estimators. Journal of Statistical Software, 16(9), 1–16.

    Article  Google Scholar 

Download references

Acknowledgments

The author wishes to thank the associate editor and two anonymous referees for their numerous comments and suggestions. David Magis is Research Associate of the Fonds de la Recherche Scientifique – FNRS (Belgium). This research was funded in part by the Research Fund of KU Leuven, Belgium (GOA/15/003) and the Interuniversity Attraction Poles program financed by the Belgian government (IAP/P7/06).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to David Magis.

Appendix: Proof of Result 1

Appendix: Proof of Result 1

To sketch the proof of Result 1, we recall first three important theorems and we introduce two technical lemmas.

The first result is known as Chebyshev’s theorem (see e.g., Rao 1984, p. 57), that is, a weak form of the law of large numbers.

Chebyshev’s theorem

if \(Y_i \,(i=1, \ldots , n)\) is a sequence of independent random variables and if \(S_n = {\sum }_{i=1}^n Y_i \) is such that

$$\begin{aligned} \frac{V(S_n )}{n^{2}} \rightarrow 0 \quad as\quad n\rightarrow \infty , \end{aligned}$$
(27)

(where \(V(.)\) stands for the mathematical variance), then \(S_n / n\) converges in probability to its expected value \(E(S_n / n) \) (with \(E(.)\) standing for the mathematical expectation). Condition (27) is referred to as Chebyshev’s condition.

The first technical lemma sets that Chebyshev’s condition is fulfilled for a particular choice of the random variables \(Y_i \).

Lemma 1

Using the notations of (2) and (3), set

$$\begin{aligned} Y_i =g_{i}^{\prime }\left( {\theta , n} \right) =a_{i}^{\prime }\left( \theta \right) X_i +b_{i}^{\prime }(\theta ,n). \end{aligned}$$
(28)

Then, under assumptions (a) to (d), the sequence of random variables \(Y_i \, (i=1, \ldots , n)\) fulfills Chebyshev’s condition. In other words, \(S_n /n=({\sum }_{i=1}^n Y_i )/n\) converges in probability toward

$$\begin{aligned} c_n =\frac{1}{n} {\sum }_{i=1}^n E(g_{i}^{\prime }\left( {\theta , n} \right) )=\frac{1}{n} {\sum }_{i=1}^n \left\{ {a_{i}^{\prime }\left( \theta \right) P_i \left( \theta \right) +b_{i}^{\prime }(\theta ,n)} \right\} . \end{aligned}$$
(29)

Moreover, the limit \(c_n \) is bounded away from zero.

Proof

First, the \(Y_i \) variables defined by (28) are independent according to assumption (a), and thus,

$$\begin{aligned} V\left( {S_n } \right) ={\sum }_{i=1}^n V[g_{i}^{\prime }\left( {\theta , n} \right) ]= {\sum }_{i=1}^n a_{i}^{\prime }\left( \theta \right) ^{2}P_i \left( \theta \right) Q_i \left( \theta \right) . \end{aligned}$$
(30)

By assumptions (b) and (d), one can directly notice that \(V\left( {S_n } \right) \le n c^{**}\) and

$$\begin{aligned} \frac{V\left( {S_n } \right) }{n^{2}}=\frac{1}{n}{\sum }_{i=1}^n \frac{a_{i}^{\prime }\left( \theta \right) ^{2}P_i \left( \theta \right) Q_i \left( \theta \right) }{n}\le \frac{c^{**}}{n}, \end{aligned}$$
(31)

which is sufficient to establish that \(V\left( {S_n } \right) / n^{2}\) converges toward zero with increasing \(n\)l, and thus, that Chebyshev’s condition is fulfilled with this choice (28) of \(Y_i \) variables. The derivation of \(c_n \) is straightforward, and assumptions (b) and (c) ensure that it is bounded away from zero. \(\square \)

The second important result is known as Liapunov’s theorem (e.g., Rao 1984, p. 283–286), which is a weak version of the central limit theorem.

Liapunov’s theorem

Let \(Y_i \,(i=1, \ldots , n)\) be a sequence of independent random variables with mean \(\mu _i \) and variance \(\sigma _i^2 \), and set \(S_n = {\sum }_{i=1}^n Y_i \) and \(s_n^2 = {\sum }_{i=1}^n \sigma _i^2 \). If it exists \(\delta >0\) such that

$$\begin{aligned} \frac{1}{s_n^{2+\delta } } {\sum }_{i=1}^n E\left[ {\left| {Y_i -\mu _i } \right| ^{2+\delta }} \right] =\frac{\mathop \sum \nolimits _{i=1}^n E\left[ {\left| {Y_i -\mu _i } \right| ^{2+\delta }} \right] }{\left[ \sum \nolimits _{i=1}^n \sigma _i^2 \right] ^{(2+\delta )/2}}\rightarrow 0\quad as \quad n\rightarrow \infty , \end{aligned}$$
(32)

then \([S_n -E(S_n )]/\sqrt{V(S_n )}\) converges in distribution to a standard normal random variable. Condition (32) is referred to as Liapunov’s condition.

The second technical lemma is a particular application of Liapunov’s theorem.

Lemma 2

Using the notations of (2) and (3), set

$$\begin{aligned} Y_i =g_i \left( {\theta , n} \right) . \end{aligned}$$
(33)

Then, under assumptions (a) to (d), the sequence of random variables \(Y_i \, (i=1, \ldots , n)\) fulfills Liapunov’s condition with \(\delta =2\). In other words, \([S_n / \sqrt{n}-E(S_n )/ \sqrt{n}]/\sqrt{V(S_n )/n}\) converges in distribution to a standard normal random variable, or equivalently the asymptotic variance of \(S_n / \sqrt{n}=({\sum }_{i=1}^n Y_i )/\sqrt{n}\) is equal to

$$\begin{aligned} v_n =\frac{1}{n} {\sum }_{i=1}^n V[g_i \left( {\theta , n} \right) ]=\frac{1}{n} {\sum }_{i=1}^n a_i \left( \theta \right) ^{2}P_i \left( \theta \right) Q_i \left( \theta \right) . \end{aligned}$$
(34)

Proof

First,

$$\begin{aligned} \mu _i =E(Y_i )=a_i \left( \theta \right) P_i \left( \theta \right) +b_i (\theta ,n) \end{aligned}$$
(35)

and

$$\begin{aligned} s_n ^{2}={\sum }_{i=1}^n V(Y_i )={\sum }_{i=1}^n a_i \left( \theta \right) ^{2}P_i \left( \theta \right) Q_i \left( \theta \right) . \end{aligned}$$
(36)

It follows that \(Y_i -\mu _i =a_i \left( \theta \right) [X_i -P_i \left( \theta \right) ]\), and hence,

$$\begin{aligned} E\left[ {\left| {Y_i -\mu _i } \right| ^{2+\delta }} \right] =a_i \left( \theta \right) ^{4} E([X_i -P_i \left( \theta \right) ]^{4}) \end{aligned}$$
(37)

with \(\delta =2\). One can rewrite then, using (36) and (37),

$$\begin{aligned} \frac{1}{s_n^{2+\delta } } {\sum }_{i=1}^n E\left[ {\left| {Y_i -\mu _i } \right| ^{2+\delta }} \right] = \frac{\mathop \sum \nolimits _{i=1}^n a_i \left( \theta \right) ^{4} E\left( {[X_i -P_i \left( \theta \right) ]^{4}} \right) \big / n^{2}}{\left[ \mathop \sum \nolimits _{i=1}^n a_i \left( \theta \right) ^{2}P_i \left( \theta \right) Q_i \left( \theta \right) \big / n\right] ^{2}}. \end{aligned}$$
(38)

Now, by assumptions (b) to (d), the function \(a_i \left( \theta \right) ^{2}P_i \left( \theta \right) Q_i \left( \theta \right) \) is upper bounded and bounded away from zero for any item, so that the denominator of (38) is also upper bounded and bounded away from zero for any \(n\). Moreover, by definition, \([X_i -P_i \left( \theta \right) ]^{4}\) takes value \(Q_i \left( \theta \right) ^{4}\) with probability \(P_i \left( \theta \right) \) and value \(P_i \left( \theta \right) ^{4}\) with probability \(Q_i \left( \theta \right) \), whence

$$\begin{aligned} E\left( {[X_i -P_i \left( \theta \right) ]^{4}} \right) =P_i \left( \theta \right) Q_i \left( \theta \right) [P_i \left( \theta \right) ^{3}+Q_i \left( \theta \right) ^{3}]. \end{aligned}$$
(39)

In sum, the numerator of (38) is equal to

$$\begin{aligned} \frac{1}{n^{2}} {\sum }_{i=1}^n a_i \left( \theta \right) ^{4} P_i \left( \theta \right) Q_i \left( \theta \right) [P_i \left( \theta \right) ^{3}+Q_i \left( \theta \right) ^{3}]\le \frac{[c^{*}]^{4}}{n} , \end{aligned}$$
(40)

according to assumptions (b) and (d). The inequality in (40) is sufficient to ensure that the numerator of (38) converges toward zero as \(n\) increases. Altogether, Liapunov’s condition is satisfied with this specific choice (33) of \(Y_i \) variables. The asymptotic variance (34) is obtained from a straightforward calculation. \(\square \)

The third important theorem to recall here is

Slutsky’s theorem

if the sequences of random variables \(X_i \) and \(Y_i \, (i=1, \ldots , n)\) are such that \(X_i \) converges in probability to some constant \(c\) and \(Y_i \) converges in distribution to the random variable \(Y\), then the sequence \(X_i Y_i \) converges to the random variable \(cY\).

We can now sketch the proof of Result 1.

Proof of Result 1

Assume that items 1 to \(n\) are administered to a respondent with true ability level \(\theta _0 \). Start by writing the difference \(\hat{\theta }-\theta _0 \) using Taylor series expansion of \(G_n \big ( {\hat{\theta }} \big )\), limited to the first order (as already pointed out, this limitation yields approximate ASE values):

$$\begin{aligned} G_n \big ( {\hat{{\theta }}} \big )=G_n \big ( {\theta _0 } \big )+\big ( {\hat{{\theta }}-\theta _0 } \big ) G_n^{\prime }\big ( {\theta _0 } \big ), \end{aligned}$$
(41)

with \(G_n^{\prime }\big ( {\theta _0 } \big )\) being the first derivative of \(G_n \big ( \theta \big )\) with respect to \(\theta \) and evaluated at \(\theta _0 \). Equation (41) can be rewritten as

$$\begin{aligned} \sqrt{n} \big ( {\hat{{\theta }}-\theta _0 } \big )=-G_n^{\prime }\left( {\theta _0 } \right) ^{-1} \sqrt{n} G_n \big ( {\theta _0 } \big ), \end{aligned}$$
(42)

provided that \(G_n^{\prime }\big ( {\theta _0 } \big )\) is not zero and since \( G_n \big ( {\hat{\theta }} \big )=0\) by definition of the ability estimator.

Now, by definition,

$$\begin{aligned} G_n^{\prime } \left( {\theta _0 } \right) =\frac{1}{n}{\sum }_{i=1}^n g_{i}^{\prime }\left( {\theta _0 , n} \right) =\frac{1}{n}{\sum }_{i=1}^n Y_i , \end{aligned}$$
(43)

with \(Y_i \) as defined in Lemma 1. Thus, according to this Lemma, (43) converges in probability to the constant \(c_n \) set by (29). Moreover,

$$\begin{aligned} \sqrt{n} G_n \left( {\theta _0 } \right) =\frac{1}{\sqrt{n}}{\sum }_{i=1}^n g_i \left( {\theta _0 , n} \right) =\frac{1}{\sqrt{n}}{\sum }_{i=1}^n Y_i , \end{aligned}$$
(44)

with \(Y_i \) as defined in Lemma 2. According to this Lemma, (44) converges in distribution to a normal random variable with variance \(\nu _n \) given by (34).

Eventually, applying Slutsky’s theorem to the right-hand side of (42), using results from (43) and (44), it is established that \(\sqrt{n} \big ( {\hat{\theta }-\theta _0 } \big )\) converges in distribution to a normal random variable with variance given by \(c_n ^{-2} v_n \), or equivalently, that the (approximate) ASE of the ability estimator is

$$\begin{aligned} \sqrt{\frac{v_n }{n \,c_n ^{2}}}=\frac{\sqrt{\sum \nolimits _{i=1}^n a_i \big ( {\theta _0 } \big )^{2}P_i \big ( {\theta _0 } \big ) Q_i \big ( {\theta _0 } \big )}}{\left| {\sum \nolimits _{i=1}^n \left\{ {a_{i}^{\prime }\big ( {\theta _0 } \big )P_i \big ( {\theta _0 } \big )+b_{i}^{\prime }(\theta _0 ,n)} \right\} } \right| }, \end{aligned}$$
(45)

and can be estimated by plugging-in the ability estimator \(\hat{\theta }\) instead of the true ability level \(\theta _0 \):

$$\begin{aligned} \widehat{ASE} \big ( {\hat{{\theta }}} \big )=\frac{\sqrt{\mathop \sum \nolimits _{i=1}^n a_i \big ( {\hat{{\theta }}} \big )^{2}P_i \big ( {\hat{{\theta }}} \big ) Q_i \big ( {\hat{{\theta }}} \big )}}{\left| {\mathop \sum \nolimits _{i=1}^n \left\{ {a_{i}^{\prime }\big ( {\hat{{\theta }}} \big )P_i \big ( {\hat{{\theta }}} \big )+b_{i}^{\prime }(\hat{{\theta }},n)} \right\} } \right| }, \end{aligned}$$
(46)

which corresponds to (16). \(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Magis, D. Efficient Standard Error Formulas of Ability Estimators with Dichotomous Item Response Models. Psychometrika 81, 184–200 (2016). https://doi.org/10.1007/s11336-015-9443-3

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-015-9443-3

Keywords

Navigation