Supporting Diagnostic Inferences Using Significance Tests for Subtest Scores

Abstract

Users of content-heterogeneous assessments based on unidimensional trait models often request information about examinee strengths and weaknesses in specific subareas. This is commonly called diagnostic information, and a standard way of providing it is by computing and reporting subscores. However, in the many cases where subscores fail to provide reliable information sufficiently independent of the total score, they cannot support claims about subarea strengths and weaknesses relative to total score expectations. These kinds of claims are referred to here as diagnostic inferences. This paper introduces a method to support diagnostic inferences for assessment programs developed and maintained using item response theory (IRT). The method establishes null and alternative hypotheses for the number correct on subsets of items or subtests. Statistical significance testing is then conducted to determine the strength of the statistical evidence in favor of a diagnostic inference. If the subtest score is modeled as a Poisson binomial distribution with probabilities set to those expected by the IRT model conditional on fixed item parameters and person scores, then a determination can be made, by individual or groups, whether and which diagnostic inferences are supported. This paper presents results of power computations showing the subtest lengths generally required for supporting diagnostic inferences under different conditions and effect sizes.