Analyses of Model Fit and Robustness. A New Look at the PISA Scaling Model Underlying Ranking of Countries According to Reading Literacy

Abstract

This paper addresses methodological issues that concern the scaling model used in the international comparison of student attainment in the Programme for International Student Attainment (PISA), specifically with reference to whether PISA’s ranking of countries is confounded by model misfit and differential item functioning (DIF). To determine this, we reanalyzed the publicly accessible data on reading skills from the 2006 PISA survey. We also examined whether the ranking of countries is robust in relation to the errors of the scaling model. This was done by studying invariance across subscales, and by comparing ranks based on the scaling model and ranks based on models where some of the flaws of PISA’s scaling model are taken into account. Our analyses provide strong evidence of misfit of the PISA scaling model and very strong evidence of DIF. These findings do not support the claims that the country rankings reported by PISA are robust.

Appendices

Appendix A. Information on Countries

Table A.1 provides information on (a) average scores and the number of students with complete responses to 20 items, (b) DIF equated scores with Azerbaijan as reference country (Kreiner and Christensen 2007), and (c) overall tests of fit of the Rasch model in 56 countries.

Table A.1. Average total and DIF equated scores on 20 items in 56 countries and conditional likelihood ratio (CLR) tests of the Rasch model comparing item parameters estimated for student with raw scores below and above the median raw score in the country.

Appendix B. Analyses of Data from All Booklets with Reading Items

In addition to Booklet 6 with 28 reading items, reading items can also be found in Booklets 2, 7, 9, 11, 12, and 13. Each of these booklets contained 14 items. Booklets 9, 11, and 13 had items from reading units R055, R104, and R111. We refer the these booklets together with Booklet 6 that also had these reading units as Booklet set 1. Booklet set 2 consists of Booklets 2, 6, 7, and 12 with reading units R067, R102, R219, and R220.

Table B.1 shows the overall CLR tests of the Rasch model for the two different booklet sets as a whole and for the different booklets. In addition to the CLR tests not DIF relative to country, CLR tests also provided evidence of DIF relative to booklets (Booklet set 1: CLR=1669.0, df=42, p<0.00005; Booklet set 2: CLR=3260.3, df=27p<0.00005).

Additional information on these analyses is available from the authors on request.

Table B.1. Overall fit statistics for Booklets sets 1 and 2 and for Booklets 2,7,9,11,12,13.

Appendix C. Assessment of Ranking Error

Let Y _cvi be the score on item i by person v from country c (c=1,…,C; v=1,…,N _c ; i=1,…,I) and let A be the indices of a subset of items A⊂{1,…,I}. The total score on all items is $S_{cv} = \sum_{i = 1}^{I} Y_{cvi}$ and the subscore over items in A is T _cv =∑ _i∈A Y _cvi . This Appendix is concerned with errors when countries are ranked according to averages $S_{c} = \frac{1}{N_{c}}\sum_{v} S_{cv}$ and $T_{c} = \frac{1}{N_{c}}\sum_{v} T_{cv}$.

We assume that item responses fit a Rasch model with a latent variable Θ. The distribution of Θ may be nonparametric or parametric. In the nonparametric case, the population parameters of interest are the score probabilities P(S _cv =s) and P(T _cv =t). The marginal distribution of T _cv is given by P(T _c =t)=∑ _s P(T _cv =t|S _cv =s)P(S _cv =s). Under the Rasch model, P(T _cv =t|S _cv =s) depends on item parameters, but not on Θ. Under such a model, it is consequently easy to calculate estimates of the subscore probabilities P(T _cv =t) in the nonparametric case if consistent estimates of the item parameters and the score probabilities P(S _cv =s) are available.

In the parametric case, we assume that the latent variables are Gaussian normal with means ξ _c and standard deviations σ _c . Given these distributions, Monte Carlo methods provide simple estimates of the distributions of S _cv and T _cv based on estimates of item parameters together with estimates of ξ _c and σ _c .

The country ranks according to (S ₁,…,S _C ) and (T ₁,…,T _C ) are expected to be similar under the Rasch model, but ranking errors will occur depending on the number of items and on sample sizes in different countries: the smaller the sample size and the smaller the number of items, the larger the ranking error. And, of course, the ranking errors also depend on ξ _c and σ _c . The results reported in this paper are derived under the parametric model with Monte Carlo estimates of the distributions of S _cv and T _cv based on Monte Carlo samples of 10,000 random students from each country.

To estimate the distribution of the country ranks based on country averages S _c and/or T _c , we generated 100,000 random values of S _c and/or T _c and for each set ranked the countries according to these values. Given these estimates, it is easy to find both confidence intervals and probabilities of extreme rankings for the countries.