Evaluating measurement equivalence using the item response theory log-likelihood ratio (IRTLR) method to assess differential item functioning (DIF): applications (with illustrations) to measures of physical functioning ability and general distress

Abstract

Background Methods based on item response theory (IRT) that can be used to examine differential item functioning (DIF) are illustrated. An IRT-based approach to the detection of DIF was applied to physical function and general distress item sets. DIF was examined with respect to gender, age and race. The method used for DIF detection was the item response theory log-likelihood ratio (IRTLR) approach. DIF magnitude was measured using the differences in the expected item scores, expressed as the unsigned probability differences, and calculated using the non-compensatory DIF index (NCDIF). Finally, impact was assessed using expected scale scores, expressed as group differences in the total test (measure) response functions. Methods The example for the illustration of the methods came from a study of 1,714 patients with cancer or HIV/AIDS. The measure contained 23 items measuring physical functioning ability and 15 items addressing general distress, scored in the positive direction. Results The substantive findings were of relatively small magnitude DIF. In total, six items showed relatively larger magnitude (expected item score differences greater than the cutoff) of DIF with respect to physical function across the three comparisons: “trouble with a long walk” (race), “vigorous activities” (race, age), “bending, kneeling stooping” (age), “lifting or carrying groceries” (race), “limited in hobbies, leisure” (age), “lack of energy” (race). None of the general distress items evidenced high magnitude DIF; although “worrying about dying” showed some DIF with respect to both age and race, after adjustment. Conclusions The fact that many physical function items showed DIF with respect to age, even after adjustment for multiple comparisons, indicates that the instrument may be performing differently for these groups. While the magnitude and impact of DIF at the item and scale level was minimal, caution should be exercised in the use of subsets of these items, as might occur with selection for clinical decisions or computerized adaptive testing. The issues of selection of anchor items, and of criteria for DIF detection, including the integration of significance and magnitude measures remain as issues requiring investigation. Further research is needed regarding the criteria and guidelines appropriate for DIF detection in the context of health-related items.

Appendix

Illustration: Calculation of Boundary and Category Response Functions, Expected Item and Scale Scores and Non-compensatory Differential Item Functioning (DIF) Indices: Polytomous items with ordinal response categories

This appendix is an illustration of the calculation of several indices used in determining the presence, magnitude and impact of DIF using item response theory. Illustrations can also be found in Collins et al. [40]; Orlando-Edelen et al. [31]; Thissen et al. [38]; and Thissen [13, 14].

Boundary Response Functions: The Samejima [25] graded response model, which assumes ordinal categories can be used to model polytomous items (see also Cohen et al. [43]. The model is based on calculation of a series of cumulative dichotomies resulting in cumulative probabilities of responding in a category or higher. One models the probability that a randomly selected individual with a specific level of physical functioning will respond in category k or higher. The boundary response function defines the cumulative probability of scoring in category k or higher:

$$ \hbox{P}_{\rm ik}(\theta ) = 1/\{1+\exp[-\hbox{a}_{\rm i}(\theta -\hbox{b}_{\rm ik})]\}. $$

There are k−1 such dichotomies. For a three category item, scored 0,1,2: the first cumulative dichotomy is between people who have a zero response vs those who selected response 1 or 2. The second cumulative dichotomy is between those who selected 0 or 1 vs 2 and higher. To illustrate using the example shown in Fig. 2, calculations are presented for θ =  −1.0.

The probability of category 0 or higher = 1 (because every one scores either 0 or higher).

For category 1: P(x = 1 or higher) =  1/{1 + exp[−a(θ −b_k1)]}

For Whites: P(x = 1 or higher) =  1/{1 + exp[−3.53((−1)−(−.90))]} =  .4127

For African-Americans: P(x = 1 or higher) =  1/{1 + exp[−2.64((−1)−(−1.19))]} =  .6228

For category 2:

P(x = 2 or higher (there is no higher)) =  1/1 + exp[−a(θ)−b_k2)]

For Whites: P(x = 2 or higher) =  1/{1 + exp[−3.53((−1)−(−.07))]} =  .0360

For African-Americans: P(x = 2 or higher) =  1/{1 + exp[−2.64((−1)−(−.01))]} =  .0683

Category response functions: The above formula does not give the probability of responding in a specific category; to get this probability, the adjacent probability is subtracted out. Note that boundary response functions are usually used because they have a consistent form across levels; category response functions do not, and are more difficult to compare and interpret. Note also that when an item is binary, the category response function for the second category (for an item coded 0,1 this is 1) is the same as the item response (boundary response) function.

To obtain the category response P(x = k), subtract out the probability that P is in a higher category: P(k)−P(k+1).

For k = 0   P_i0(θ) = [P_i0(θ)−P_i1(θ)] =  [1−P_i1(θ)]

For k = 1   P_i1(θ) = [P_i1(θ)−P_i2(θ)]

For k = 2   P_i2(θ) = [P_i2(θ)− 0]

For ${\varvec{\theta} {\bf =-1.0:}}$

For Whites: P_i0 =  1 − .4127 =  .5873

For African-Americans: P_i0 =  1 − .6228 =  .3772

For Whites: P_i1(θ = −1.0) = .4127 − .0360 =  .3767

For African-Americans: P_i1(θ = −1.0) = .6228 − .0683 =  .5545

For Whites: P_i2(θ = −1.0) = .0360

For African-Americans: P _i2(θ = −1.0) = .0683

The shapes of the CRFs are not the same, because they are no longer cumulative. A person with a specific theta level will have a separate probability of response for each response category. This category response function provides the probability that a randomly selected individual at say θ = 0 (average ability) will respond in category k.

Computing expected item and test scores: Expected item and test (scale) scores can be computed for both dichotomous and polytomous items. Lord and Novick and Birnbaum [24, p 386] introduce the notion of true scores in the context of IRT. A person’s true score is their expected score, expressed in terms of probabilities for binary items and in terms of weighted probabilities for polytomous items. The test characteristic curve described in Lord and Novick related true score or averaged expected test score to theta.

For a dichotomous item scored 0 and 1, the expected or true score is simply the probability of scoring in the ‘1’ category, given an individual’s estimated ability or P_i(θ_s), where P _i (θ_s) is the probability of scoring a ‘1’ on item i for subject s.

$$ \hbox{P}_{\rm i}(\theta_{\rm s}) = 1/\{1+\exp[-\hbox{a}_{\rm i}(\theta _{\rm s}-\hbox{b}_{\rm i})]\}. $$

(Here it is assumed that c (guessing) parameters are estimated at 0).

For a polytomous item, taking a graded response form, the expected score is the sum of the weighted probabilities of scoring in each of the possible categories for the item. For an item with 5 response categories, coded 0 to 4, for example, this sum would be:

$$ 0 \ast[\hbox{P}_{\rm i0}(\theta _{\rm s})] + 1 \ast \hbox{P}_{\rm i1}(\theta _{\rm s}) + 2 \ast \hbox{P}_{\rm i2}(\theta _{\rm s})+ 3 \ast \hbox{P}_{\rm i3}(\theta _{\rm s})+ 4 \ast \hbox{P}_{\rm i4}(\theta _{\rm s}) $$

For the graded response item with k categories, there are k−1 estimated bs and so in the example above, there will be four probabilities computed.

Recall that in the graded response model, the boundary response function defines the cumulative probability of scoring in category k or higher:

$$ \hbox{P}_{\rm ik}(\theta _{\rm s}) = 1/\{1+\exp[-\hbox{a}_{\rm i}(\theta _{\rm s} -\hbox{b}_{\rm ik})]\}. $$

Thus, the probability of scoring above category k must be subtracted out in order to obtain the probability of scoring in the category. (This was illustrated in the previous section.)

Note that P_i(θ_s) =  1/{1 + exp[−a_i(θ_s−b_i1)]} − 1/{1 + exp[−a_i(θ_s−b_i2)]}.

As an example, the expected or true score as a member of the African-American group is the sum of the weighted probabilities of scoring in each of the possible categories for each θ, coded 0,1,2.

$$ 0 + 1 \ast \hbox{P}_{\rm i1}(\theta )+ 2 \ast \hbox{P}_{\rm i1}(\theta) $$

A true or expected score for an individual of mild disability (θ = −1.0) as a member of the white group would be:

$$ 0 (.59) + 1(.3767) + 2(.0360) = .4487 $$

A true or expected score for an individual of mild disability (θ = −1.0) as a member of the African-American group would be:

$$ 0 + 1 (.5545) + 2 (.0683) = .6905 $$

The expected test score for a subject with estimated ability θ is simply the sum of the expected item scores for that individual. Plots of expected scores against theta can then be constructed for given values of theta. Individual expected scores are used in the calculation of magnitude and impact indices discussed below.

Computing Non-Compensatory Differential Item Functioning (NCDIF): Two measures developed by Raju and colleagues [16] are based on IRT (see also Flowers et al. [17]). These measures are compensatory and non-compensatory DIF, or CDIF and NCDIF. NCDIF is more like indices of DIF such as the area statistics and Lord’s chi-square; the assumption is that all other items in the test are unbiased, except for the studied item. CDIF does not make this assumption. The advantage of these measures over Lord’s chi-square and the area statistics, such as Raju’s signed and unsigned area statistic is that they are based on the actual distribution of the ability estimates within the group for which it is desired to estimate bias, rather than the entire theoretical range of theta. If, for instance, most members of the focal group fall within the range of theta from −1 to 0, rather than between −1 to +1 on the continuum, the area statistics will give an inaccurate estimate of DIF.

NCDIF is computed exactly like the unsigned probability difference of Camilli and Shepard [15]. For each subject in the focal group, two estimated scores are computed. One is based on the subject’s ability estimate and the estimated a, b and c parameters for the focal group, and the other based on the ability estimate and the estimated a, b and c parameters for the reference group. Each subject’s difference score (d) is squared, and these squared difference scores are added for all subjects (j = 1, n) to obtain NCDIF.

$$ \begin{array}{l} \hbox{NCDIF}_{\rm i} = [\displaystyle \sum_{j=1,n} (\hbox{ES}_{\rm siF }- \hbox{ES}_{\rm siR })^{2 }]\\ \end{array} $$

As an example, NCDIF for item i is the average difference squared between the true or expected scores for an individual (s) as a member of the focal group (F) and as a member of the reference group (R). Using the example shown above for a person at theta=-1.0,

$$ \hbox{d} = (.6905 - .4487)^{2} = .0585 $$

This quantity is then summed across people and averaged ∑d / n. This value is the NCDIF, which (as mentioned above) is also the unsigned probability difference (UPD) illustrated by Camilli and Shepard [15]. NCDIF is the average difference between the true (expected) scores for groups, and provides a measure of DIF magnitude. New methods for determining NCDIF cutoffs for binary items have recently been described [44].