New Developments in Quantitative Psychology pp 235-250
Evaluating CTT- and IRT-Based Single-Administration Estimates of Classification Consistency and Accuracy
- First Online:
- Cite this paper as:
- Deng N., Hambleton R.K. (2013) Evaluating CTT- and IRT-Based Single-Administration Estimates of Classification Consistency and Accuracy. In: Millsap R., van der Ark L., Bolt D., Woods C. (eds) New Developments in Quantitative Psychology. Springer Proceedings in Mathematics & Statistics, vol 66. Springer, New York, NY
The percentage of examinees who are classified consistently and accurately into the proficiency levels is an important measurement property of the tests that are used to classify the candidates. Given the suspected discrepancies between the classical test theory (CTT)- and item response theory (IRT)-based single-administration decision consistency and accuracy (DC/DA) estimates, these two approaches were evaluated for accuracy and robustness in various simulated conditions by varying the test length, ability distribution, and the degree of local item dependence (LID). The CTT-based Livingston–Lewis method was found underestimating the DC indices across all conditions and more sensitive to the short tests and skewed ability distributions. The IRT-based Lee method had small biases in most conditions except a high degree of LID. The violation of LID had a much greater negative effect on the DA estimate than on the DC estimate with both methods.