Chapter

New Developments in Quantitative Psychology

Volume 66 of the series Springer Proceedings in Mathematics & Statistics pp 235-250

Date:

Evaluating CTT- and IRT-Based Single-Administration Estimates of Classification Consistency and Accuracy

  • Nina DengAffiliated withDepartment of Quantitative Health Sciences, University of Massachusetts Medical School Email author 
  • , Ronald K. HambletonAffiliated withCenter for Educational Assessment, University of Massachusetts Amherst

* Final gross prices may vary according to local VAT.

Get Access

Abstract

The percentage of examinees who are classified consistently and accurately into the proficiency levels is an important measurement property of the tests that are used to classify the candidates. Given the suspected discrepancies between the classical test theory (CTT)- and item response theory (IRT)-based single-administration decision consistency and accuracy (DC/DA) estimates, these two approaches were evaluated for accuracy and robustness in various simulated conditions by varying the test length, ability distribution, and the degree of local item dependence (LID). The CTT-based Livingston–Lewis method was found underestimating the DC indices across all conditions and more sensitive to the short tests and skewed ability distributions. The IRT-based Lee method had small biases in most conditions except a high degree of LID. The violation of LID had a much greater negative effect on the DA estimate than on the DC estimate with both methods.