The practical impact of differential item functioning analyses in a health-related quality of life instrument

Abstract

Introduction

Differential item functioning (DIF) analyses are commonly used to evaluate health-related quality of life (HRQoL) instruments. There is, however, a lack of consensus as to how to assess the practical impact of statistically significant DIF results.

Methods

Using our previously published ordinal logistic regression DIF results for the Fatigue scale of a HRQoL instrument as an example, the practical impact on a particular Norwegian clinical trial was investigated. The results were used to determine the difference in mean Fatigue scores assuming that the same trial was conducted in the UK. The results were then compared with published information on what would be considered a clinically important change in scores.

Results

The item with the largest DIF effect resulted in differences between the mean English and Norwegian Fatigue scores that, although small, could be considered clinically important. Sensitivity analyses showed that larger differences were found for shorter scales, and when the proportions in each response category were equal.

Discussion

Our scenarios suggest that translation differences in an item can result in small, but clinically important, differences at the scale score level. This is more likely to be problematic for observational studies than for clinical trials, where randomised groups are stratified by centre.

Appendix

Coding the four response categories 0, 1, 2 and 3, respectively, the following three equations apply when conducting ordinal logistic regression using the proportional odds model:

$$ {\text{logit(}}\Pr (Y \ge j ) )= \beta_{{0_{j} }} + \beta_{1}^{*} {\text{LANG}} + \beta_{2}^{*} {\text{FA}} + \beta_{3}^{*} {\text{AGE}} + \cdots ,\quad \, j = 1,2,3 $$

(1)

where LANG is the language/translation (0 for English, 1 for Norwegian), FA is the overall Fatigue scale score and adjustment is also made for age (AGE) and other covariates.

These equations may then be written out separately for Norwegian (NO) and English (EN) speakers:

$$ \begin{array}{*{20}c} {{\text{logit}}(\Pr (Y_{\text{NO}} \ge j )) = \beta_{{0_{j} }} + \beta_{1} + \beta_{2}^{*} {\text{FA}} + \beta_{3}^{*} {\text{AGE}} + \cdots ,\quad \, j = 1,2,3} \hfill \\ {{\text{logit}}(\Pr (Y_{\text{EN}} \ge j )) = \beta_{{0_{j} }} + \beta_{2}^{*} {\text{FA}} + \beta_{3}^{*} {\text{AGE}} + \cdots ,\quad \, j = 1,2,3} \hfill \\ \end{array} $$

(2)

Combining the two equations and rearranging gives

$$ {\text{Pr(}}Y_{\text{EN}} \ge \, j )= {\frac{1}{{\left( {{\frac{{1 - {\text{Pr(}}Y_{\text{NO}} \ge \, j )}}{{{\text{Pr(}}Y_{\text{NO}} \ge \, j )}}}} \right)e^{{\beta_{1} }} + 1}}},\quad j = 1,2,3 $$

(3)

From the results of our DIF analyses for Q18, the estimate of β₁ was found to be −1.089. From Table 2, for this particular study:

$$ \begin{aligned} &{\Pr (Y_{\text{NO}} \ge 1) = 450/513 = 0.877}\\ &{\Pr (Y_{\text{NO}} \ge 2) = 242/513 = 0.472} \\ &{\Pr (Y_{\text{NO}} \ge 3) = 87/513\;\;= 0.170} \end{aligned} $$

(4)

Substituting these values into formulae [3] gives:

$$ \begin{array}{*{20}c} {\Pr (Y_{\text{EN}} \ge \, 1) = 0.955} \\ {\Pr (Y_{\text{EN}} \ge \, 2) = 0.726} \\ {\Pr (Y_{\text{EN}} \ge \, 3) = 0.378} \\ \end{array} $$

(5)

Hence the proportions choosing each category can be deduced, assuming that the same study was conducted using English-speaking patients:

$$ \begin{array}{ll} {{\text{Pr(not at all}}_{\text{EN}} )= 1- 0. 9 5 5= 0.0 4 5} \\ {{\text{Pr(a little}}_{\text{EN}} )= 0. 9 5 5- 0. 7 2 6= 0. 2 2 9} \\ {{\text{Pr(quite a bit}}_{\text{EN}} )= 0. 7 2 6- 0. 3 7 8= 0. 3 4 8} \\ {{\text{Pr(very much}}_{\text{EN}} )= 0. 3 7 8} \\ \end{array} $$

(6)

By comparison with the proportions in Table 2, this would mean that English speakers would be more likely to score highly on this item than Norwegian speakers.

Assuming that Q18 is the only item with true DIF, how would this affect the mean Fatigue scale scores of the patients in this study? Using scores of 0, 33.33, 66.67 and 100 for the four categories of Q18, the average scores for this item for Norwegian and English speakers would be:

$$ \begin{array}{*{20}c} {{\text{Norwegian: }}0 \times 0. 1 2 3+ 3 3. 3 3\times 0. 40 6+ 6 6. 6 7\times 0. 30 2+ 100 \times 0. 1 70 = 50. 6 2} \hfill \\ {{\text{English: }}0 \times 0.0 4 5+ 3 3. 3 3\times 0. 2 2 9+ 6 6. 6 7\times 0. 3 4 8+ 100 \times 0. 3 7 8= 6 8. 6 3} \hfill \\ \end{array} $$

(7)

Therefore, Norwegian speakers would be expected to score on average 68.63 − 50.62 = 18.01 points lower on this item.

The Fatigue scale score is made up of three items of equal weighting, so this would mean that for the overall scale score Norwegians would be expected to score 18.01/3 = 6.00 more than English speakers on average.

Table 1 shows that the 95% confidence limits for the language effect for Q18 (β₁) were −1.271 and −0.908. Following the same methods as for β₁ itself (working not shown), this implies that the difference between English and Norwegian speakers on the Fatigue subscale would be 6.00 (95% CI: 5.03–6.95).