Differential item functioning was negligible in an adaptive test of functional status for patients with knee impairments who spoke English or Hebrew

Abstract

Objective

We examined the presence and impact of differential item functioning (DIF) in a set of knee-specific functional status (FS) items administered using computerized adaptive testing (CAT) among English (United States) and Hebrew (Israel) speaking patients receiving therapy for knee impairments. DIF occurs in an item if probabilities of endorsing responses differ across groups after controlling for the FS measured.

Methods

We analyzed data from 28,320 patients (14,160 U.S., 14,160 Israel) who completed the knee-specific CAT. Items were assessed for DIF by gender, age, symptom acuity, surgical history, exercise history, and language spoken using a hybrid technique that combines multiple ordinal logistic regression and item response theory FS estimates.

Results

Several items had non-uniform DIF for covariates including language, but unadjusted and DIF-adjusted functional status estimates were in strong concordance [ICC(2,1) values ≥0.97], and differences between unadjusted and adjusted FS scores represented <0.4% of the unadjusted FS standard deviation.

Conclusions

Statistically significant DIF was identified in some items but represented negligible clinical impact. Results suggested no need to adjust items for DIF when assessing FS outcomes across groups of patients with knee impairments who answer the knee CAT items in English in the United States or Hebrew in Israel. These findings suggest negligible differences in cultural perceptions between English and Hebrew wording of these knee-specific CAT FS items.

Appendices

Appendix 1

Primer for using difwithpar: detailed DIF detection methods

Crane et al. [43, 47] have developed an approach to DIF assessment that combines ordinal logistic regression and IRT. Details of this approach are outlined in earlier publications.

Here, we use IRT scores initially to evaluate items for DIF. We examine three ordinal logistic regression models for each item for each demographic category (labeled here as “group”) selected for analysis:

$$ {\text{Logit}}\,P(Y = 1|\theta ,\,{\text{group}}) = \beta_{1}^{*} \theta + \beta_{2}^{*} \,{\text{group}} + \beta_{3}^{*} \theta^{*}\,{\text{group}}\quad ( {\text{model}}\, 1 ) $$

$$ {\text{Logit}}\,P(Y = 1|\theta ,\;{\text{group}}) = \beta_{1}^{*} \theta + \beta_{2}^{*} \;{\text{group}}\quad ( {\text{model}}\, 2 ) $$

$$ {\text{Logit}}\,P(Y = 1|\theta ) = \beta_{1}^{*} \theta \quad ({\text{model}}\, 3) $$

In these equations, P(Y = 1) is the probability of endorsing an item, θ is the IRT estimate of functional status, and group is the demographic category.

Two types of DIF are identified in the literature. In items with non-uniform DIF, demographic interference between ability level and item responses differs at varying levels of functional status. In items with uniform DIF, this interference is the same across all levels of functional status.

To detect non-uniform DIF, we compare the log likelihoods of models 1 and 2 using a χ² test, α = 0.05. To detect uniform DIF, we determine the relative difference between the parameters associated with θ (β₁ from models 2 and 3) using the formula |(β_{1(model 2)} − β_{1(model 3)})/β_{1(model 3)}|. If the relative difference is large, group membership interferes with the expected relationship between functional status and item responses. The literature provides little guidance regarding how large the relative difference should be. A simulation by Maldonado and Greenland on confounder selection strategies used a 10% change criterion in a very different context [61], which we followed in this data set.

Crane et al. [43, 47] have developed an approach to generate scores that account for DIF. When DIF is found, we create new datasets: items without DIF have item parameters estimated from the whole sample, while items with DIF have demographic-specific item parameters estimated.

We have written Stata code that automates these steps called difwithpar. As with runparscale, this can be obtained by typing

• .scc install difwithpar

at the Stata prompt. To evaluate items for DIF with respect to age, Stata code might look like this:

• .difwithpar dep1-dep18, id(studyid) ability(theta_naive) group(age) mul(age) runname(a01) ubch(0.10) nupv(0.001)

That code identifies the items to examine for DIF, a unique study identifier that will be used to merge scores, the ability estimate to use for conditioning, the name of the covariate to examine for DIF, the run name (“a01” here), the value of change in the β coefficient for uniform DIF detection (“ubch,” here set to the default of 0.10), and the P value for non-uniform DIF (“nupv,” here set to 0.001 to accommodate the large data set).

This single line of code will evaluate each of the items in the item list (here, dep1 through dep18) using the three ordinal logistic regression models described earlier. It will apply the criteria the user specifies (e.g., the P value for non-uniform DIF and the change in β value for uniform DIF) to flag items as having or not having DIF. For items flagged with DIF, the code will generate new items. For example, if item 3 was flagged with DIF by age, difwithpar will create items called dep3_age1, dep3_age2, and dep3_age3. The item dep3_age1 has missing values for all participants for whom age = 2 or age = 3, and the original dep3 responses for all participants for whom age = 1. The difwithpar procedure proceeds to create a Parscale dataset and Parscale code that incorporates these new items. In this example, the Parscale run would involve dep1, dep2, dep3_age1, dep3_age2, dep3_age3, dep4, dep5, dep6, dep7, … dep18. The difwithpar program then runs that code in Parscale, and imports the IRT score that accounts for DIF found in dep3, which will be called theta_a01, and its standard error, setheta_a01.

Spurious false-positive and false-negative results may occur if the functional status score (θ) used for DIF detection includes many items with DIF [43]. We therefore use an iterative approach for each covariate. We generate IRT scores that account for DIF, and use these as the FS score to detect DIF. If different items are identified with DIF, we repeat the process described earlier, modifying the assignments of items based on the most recent round of DIF detection. If the same items are identified with DIF on successive rounds, we are satisfied that we identified items with DIF (as opposed to spurious findings).

From a practical standpoint, what we do is to modify the initial line of difwithpar code, which we repeat here:

• .difwithpar dep1-dep18, id(studyid) ability(theta_naive) group(age) mul(age) runname(a01) ubch(0.1) nupv(0.001)

The modified code looks like this:

• .difwithpar dep1-dep18, id(studyid) ability(theta_a01) group(age) mul(age) runname(a02) ubch(0.1) nupv(0.001)

We have changed the ability term for DIF detection to the recently calculated theta_a01, and changed the run name to a02. We look at the items flagged with DIF in this run and compare to those found on earlier runs. We continue this process until the same items are identified with DIF on repeated runs. This may take several runs. In this example, say it took three runs to converge on the same list of items with DIF for age. The variable theta_a02 would be our estimate of functional status that accounted for DIF with respect to age.

We performed two sets of analyses. We initially examined each covariate in turn for DIF, using the initial IRT score that did not account for DIF as the conditioning variable for each covariate. We then assessed DIF impact. If DIF impact was negligible, we did not account for items with DIF simultaneously, which is what we found in our data. However, to demonstrate the effect of simultaneously controlling for DIF with all covariates regardless of DIF impact, in our final analyses, we decided to check the effect of assessing DIF simultaneously regardless of whether there was important DIF impact. Therefore, we accounted for multiple sources of DIF simultaneously. Here we describe further details of the methods for doing this.

“Appendix 5” informs us that we found one item with some form of DIF with respect to age—standing for 1 h. The final IRT estimate of FS levels included separate calibrations of this item in those 18 to 45, >45 to 65, and >65 years old. To then account for DIF according to our next covariate symptom acuity, we modify the data set to evaluate 18 items:

• 17 items that did not have DIF related to age,

• Standing among patients 18–45 years,

• Standing among patients >45–65 years,

• Standing among patients >65 years.

We have written Stata code called mergevirtual that we run after difwithpar has converged on a final answer for a covariate to bring in the new items—in this case, standing separately in the three levels of age. The code for mergevirtual is automatically loaded when difwithpar is installed. After running difwithpar with respect to age, we enter the following line at the Stata prompt:

• .mergevirtual studyid “C:\Data\mydataset”

New age-specific items for standing are then added to the data set. We proceed to analyze DIF with respect to acuity for these 20 items, and use as our conditioning variable the IRT score that accounted for DIF with respect to age.

This is operationalized by the following line of code:

• .difwithpar dep1 dep2 dep3_a1 dep3_a2 dep3_a3 dep4 dep5 dep6 dep7 dep8 dep9 dep10 dep11 dep12 dep13 dep14 dep15 dep16 dep17 dep18, id(studyid) gr(acuitygroup) mul(acuitygroup) run(all01) ab(theta_a02) ubch(0.1) nupv(0.05)

It is often useful to re-name the group-specific items (e.g., rename dep3_age1 as dep3_a1). This code tells difwithpar to use the final IRT score that accounted for DIF with respect to age (theta_a02) as the conditioning variable for DIF with respect to age group. The next run would look like this:

• .difwithpar dep1 dep2 dep3_a1 dep3_a2 dep3_a3 dep4 dep5 dep6 dep7 dep8 dep9 dep10 dep11 dep12 dep13 dep14 dep15 dep16 dep17 dep18, id(studyid) gr(acuitygroup) mul(acuitygroup) run(all02) ab(theta_all01) ubch(0.1) nupv(0.001)

Now, we have modified the previous code to use the new FS estimate—theta_all01 rather than theta_a02—and specified that the run name is all02. These steps continue until the items flagged with DIF have converged. The resulting FS estimate accounts for DIF related to age and symptom acuity. For example, if it took five runs to converge on a common list of items with DIF for acuity, the thetaall_05 term would account for DIF related to both age and acuity. Analyses then proceed by merging the new items back into the dataset, again using the code

• .mergevirtual id(studyid) “C:\Data\mydataset”

Analyses would then proceed by moving to the next covariate. For example, let’s say that DIF was found with respect to acuity for items dep1 and the previous age (18 to <45 years) versus standing item dep3_a1. Six new variables would be added to the dataset: dep1_acuitygroup1, dep1_acuitygroup2, dep1_acuitygroup3, dep3_a1_acuitygroup1, dep3_a1_acuitygroup2, and dep3_a1_acuitygroup3. Re-name the longer variable names dep1_ac1, dep1_ac2, dep1_ac3, dep3_a1_ac1, dep3_a1_ac2, and dep3_a1_ac3, and then the next difwithpar run would look like this:

• .difwithpar dep1_ac1 dep1_ac2 dep1_ac3 dep2 dep3_a1_ac1, dep3_a1_ac2, dep3_a1_ac3, dep3_a2 dep3_a3 dep4 dep5 dep6 dep7 dep8 dep9 dep10 dep11 dep12 dep13 dep14 dep15 dep16 dep17 dep18, id(studyid) gr(surgery) run(all06) ab(theta_all05) ubch(0.1) nupv(0.001)

This code tells difwithpar to analyze those items for DIF with respect to surgery group (“surgery”), conditioning on the IRT FS estimate that accounted for DIF with respect to age and acuity (“thetaall_05”). Subsequent runs proceed until all covariates of interest have been included.

The difwithpar package is capable of analyzing DIF in multiple groups. For example, our age and acuity terms included three groups. The “mul” option tells difwithpar to treat age and acuity groups as categorical group indicators. The difwithpar package will generate interaction terms between age and theta, and model 1 will look like this:

$$ {\text{Logit}}\,P(Y = 1|\theta ,{\text{agegroup}}) = \beta_{1}^{*} \theta + \beta_{2,2}^{*} \,{\text{agegroup}}_{2} + \beta_{2,3}^{*} \,{\text{agegroup}}_{3} + \beta_{2,4}^{*} \,{\text{agegroup}}_{4} + \beta_{3,2}^{*} \theta^{*}\,{\text{agegroup}}_{2} + \beta_{3,3}^{*} \theta^{*}\,{\text{agegroup}}_{3} + \beta_{3,4}^{*} \theta^{*}\,{\text{agegroup}}_{4} \quad ({\text{model}}\, 1^{*}) $$

Model 2 will then look like this:

$$ {\text{Logit}}\,P(Y = 1|\theta ,{\text{agegroup}}) = \beta_{1}^{*} \theta + \beta_{2,2}^{*} \,{\text{agegroup}}_{2} + \beta_{2,3}^{*} \,{\text{agegroup}}_{3} + \beta_{2,4}^{*} \,{\text{agegroup}}_{4} \quad ({\text{model}}\, 2^{*}) $$

And model 3 is unchanged:

$$ {\text{Logit}}\,P(Y = 1|\theta ,\,{\text{agegroup}}) = \beta_{1}^{*} \theta \quad ({\text{model}}\,3) $$

For non-uniform DIF for categorical groups, twice the differences in log likelihood of models 1* and 2* are compared to a χ² distribution with degrees of freedom equal to the number of groups minus 1. In this instance, there are four groups, and three degrees of freedom, corresponding to the three terms that are different between models 1* and 2*. For uniform DIF, the β₁ coefficient from model 2* is compared to the β₁ coefficient from model 3.

Appendix 2

See Table 3.

Table 3 Item exposure rates of the CAT in United States and Israel

Appendix 3

See Table 4.

Table 4 Item parameters by country

Appendix 4

See Table 5.

Table 5 Presence of differential item functioning related to covariates in the Israeli sample

Appendix 5

See Table 6.

Table 6 Presence of differential item functioning related to covariates in the combined United States and Israeli sample