Differential item functioning impact in a modified version of the Roland–Morris Disability Questionnaire

Abstract

Objective

To evaluate a modified version of the Roland–Morris Disability Questionnaire for differential item functioning (DIF) related to several covariates.

Background

DIF occurs in an item when, after controlling for the underlying trait measured by the test, the probability of endorsing the item varies across groups.

Methods

Secondary data analysis of two studies of participants with back pain (total n = 875). We used a hybrid item response theory/ logistic regression approach for detecting DIF. We obtained scores that accounted for DIF. We evaluated the impact of DIF on individual and group scores, and compared scores that ignored or accounted for DIF in terms of the strength of association with SF-36 subscale scores.

Results

DIF was found in 18/23 items. Salient scale-level differential functioning was found related to age, education, and employment. Overall 24 participants (3%) had salient scale-level differential functioning. Mean scores across demographic groups differed minimally when accounting for DIF. The strength of association of scores with SF-36 scores was similar for scores that ignored and scores that accounted for DIF.

Conclusions

The modified version of the Roland–Morris Disability Questionnaire appears to have largely negligible DIF related to the covariates assessed here.

Appendix 1

Detailed methods of DIF detection

We have developed an approach to DIF assessment that combines ordinal logistic regression and IRT. Details of this approach are outlined in earlier publications [14, 17]. The modified version of the Roland-Morris Disability Questionnaire contains only dichotomous items, so logistic regression was used for all DIF analyses.

We use IRT scores to initially evaluate items for DIF. We examine three models for each item for each demographic category (labeled here as “group”) selected for analysis:

$$ {\hbox{Logit}}\;p(Y = 1|\theta ,\;{\hbox{group}}) = \beta _1 *\theta + \beta _2 *{\hbox{group}} + \beta _3 *\theta *{\hbox{group}} $$

(model 1)

$$ {\hbox{Logit}}\;p(Y = 1|\theta ,\;{\hbox{group}}) = \beta _1 *\theta + \beta _2 *{\hbox{group}} $$

(model 2)

$$ {\hbox{Logit}}\;p(Y = 1|\theta ) = \beta _1 *\theta . $$

(model 3)

In these equations, p(Y = 1) is the probability of endorsing an item, θ is the IRT estimate of back pain disability, and group is the demographic category.

Two types of DIF are identified in the literature. In items with non-uniform DIF, demographic interference between ability level and item responses differs at varying levels of back pain disability. In items with uniform DIF, this interference is the same across all levels of back pain disability.

To detect non-uniform DIF, we compare the log likelihoods of models 1 and 2 using a χ² test, α = 0.05. To detect uniform DIF, we determine the relative difference between the parameters associated with θ (β₁ from models 2 and 3) using the formula $|(\beta _{1({\rm{model}}\;2)} - \beta _{1({\rm{model}}\;3)} )/\beta _{1({\rm{model}}\;3)} |.$ If the relative difference is large, group membership interferes with the expected relationship between back pain disability and item responses. There is little guidance from the literature regarding how large the relative difference should be. A simulation by Maldonado and Greenland on confounder selection strategies used a 10% change criterion in a very different context [21]. We have previously used 10% [17] and 5% [14] change criteria. In this data set, we compared results for each covariate using a 5 and 10% criterion. While there was little difference between results using a 5 and 10% criterion, we chose to show the results from the more sensitive 5% criterion.

We have developed an approach to generate scores that account for DIF [14]. When DIF is found, we create new datasets as summarized in Fig. 1. Items without DIF have item parameters estimated from the whole sample, while items with DIF have demographic-specific item parameters estimated.

Fig. 2

Impact of DIF related to eight covariates on estimated modified Roland IRT score. In this box and whisker plot, the box indicates the 25th and 75th percentiles, while the whiskers indicate 1½ times the interquartile range. Observations more extreme are indicated by dots. The graph shows the difference between scores accounting for DIF for each covariate and unadjusted scores. If there were no impact of DIF for an individual that observation would be at 0. Vertical lines placed at multiples of 0.313 and −0.313 indicate the median standard error of measurement found in this study. The three covariates for which participants have differences greater than the median standard error of measurement (age, education, and employment) are said to be associated with salient scale-level differential functioning

Spurious false-positive and false-negative results may occur if the back pain disability score (θ) used for DIF detection includes many items with DIF [2]. We therefore use an iterative approach for each covariate. We generate IRT scores that account for DIF, and use these as the back pain disability score to detect DIF. If different items are identified with DIF, we repeat the process outlined in Fig. 1, modifying the assignments of items based on the most recent round of DIF detection. If the same items are identified with DIF on successive rounds, we are satisfied that we identified items with DIF (as opposed to spurious findings).

We have modified this approach for demographic categories with more than two groups (such as education in this data set). Indicator terms for each group are generated, and interaction terms are generated by multiplying θ by the indicator terms. All indicator terms and interaction terms are included in model 1; all indicator terms are included in model 2; and only the ability term θ is included in model 3. For the determination of non-uniform DIF, we compared the likelihoods of models 1 and 2 to a χ² distribution with degrees of freedom equal to the number of groups minus 1. The determination of uniform DIF is unchanged, except all the group terms are included in model 2.