Evaluation of the link between the Guttman errors and response shift at the individual level

Abstract

Purpose

Methods for response shift (RS) detection at the individual level could be of great interest when analyzing changes in patient-reported outcome data. Guttman errors (GEs), which measure discrepancies in respondents’ answers compared to the average sample responses, might be useful for detecting RS at the individual level between two time points, as RS may induce an increase in the number of discrepancies over time. This study aims to establish the link between recalibration RS and the change in the number of GEs over time (denoted index $I$) via simulations and explores the discriminating ability of this index.

Methods

We simulated the responses of individuals affected or not affected by recalibration RS (defined as changes in the patients’ standard of measurement) to determine whether simulated individuals with recalibration had a greater change in the number of GEs over time than individuals without recalibration. The effects of factors related to the sample, the questionnaire structure and recalibration were investigated. As an illustrative example, the change in the number of GEs was computed in patients suffering from eating disorders.

Results

Within simulations, simulated individuals affected by recalibration had, on average, a greater change in the number of GEs over time than did individuals without RS. Some of the parameters related to the questionnaire structure and recalibration magnitude appeared to have substantial effects on the values of $I$. Discriminating abilities appeared, however, globally low.

Conclusion

Some evidence of the link between recalibration and the change in GEs was found in this study. GEs could be a valuable nonparametric tool for RS detection at a more individual level, but further investigation is needed.

Appendix 1: simulation implementation

Longitudinal partial credit model

The longitudinal Partial Credit Model (LPCM) was chosen to generate data since it allowed modelling response categories probabilities of polytomous items forming a unidimensional scale across time, and provided a possibility to simulate RS for a changing proportion of patients. The probability of patient $n$ to answer $m~(=0,\dots , M-1)$ on item $j$ at time $t$ under the LPCM is given by:

$$P\left({X}_{nj}^{(t)}=m|{\theta }_{n}^{(t)}, {\delta }_{j1}^{(t)},\dots , {\delta }_{jM-1}^{\left(t\right)}\right)=\frac{\mathrm{exp}\left(m{. \theta }_{n}^{\left(t\right)}-\sum_{p=1}^{m}{\delta }_{jp}^{\left(t\right)}\right)}{\sum_{l=0}^{M-1}\mathrm{exp}\left(l.{\theta }_{n}^{\left(t\right)}-\sum_{p=1}^{l}{\delta }_{jp}^{\left(t\right)}\right)}$$

where ${X}_{nj}^{(t)}$ denotes the response to the item $j=1,\dots , J$ of the individual $n$ at time $t$

${\theta }_{n}^{\left(t\right)}$ stands for the latent variable level of the individual $n$ at $t$ (realization of the random variable $\Theta$).

$$\left(\begin{array}{c}{\Theta }^{{(t}_{1})}\\ {\Theta }^{{(t}_{2})}\end{array}\right)\sim N\left(\left[\begin{array}{c}{\mu }_{1}\\ {\mu }_{2}\end{array}\right],\Sigma =\left[\begin{array}{cc}{\sigma }_{1}^{2}& {\sigma }_{\mathrm{1,2}}\\ {\sigma }_{\mathrm{1,2}}& {\sigma }_{2}^{2}\end{array}\right]\right)$$

${\delta }_{jm}^{\left(t\right)}$ is the difficulty of the response category $m=1,\dots , M-1$ from item $j$ at the time point $t$. If ${\delta }_{jm}^{\left(t\right)}$ is low, the proportion of patients scoring $m$ or more to item $j$ will be high: $m$ is hence an easy response category (vice versa for difficult response categories). Null response categories do not have a difficulty parameter.

At the first measurement occasion, difficulty parameters were chosen to be spaced along the latent variable continuum (assumed normally distributed, with a zero mean and a standard deviation equaled to 1). For each item$j$, the difficulty parameter of the first positive response category (denoted ${\delta }_{j1}^{(t_1)}$) equaled the $\frac{j}{J+1}th$ quantile from a $N\left(\mathrm{0,1}\right)$. Difficulty parameters of the following response categories were then regularly shifted from the first one: ${\delta }_{jm}^{\left({t}_{1}\right)}={\delta }_{j1}^{({t}_{1})}+\left(m-1\right)\times \frac{2}{M-2}$. Finally, difficulty parameters of all items were centered on the mean $\overline{\updelta }=\frac{\sum_{j,m}{\delta }_{jm}^{\left({t}_{1}\right)}}{J(M-1)}$ so that difficulty parameters were centered on the mean of the latent variable distribution (i.e. 0). It hence corresponded to the situation where the questionnaire is suitable for a population with a latent variable following a standard normal distribution. At the first measurement occasion, the model is a rating scale model.

Recalibration operationalization

To simulate the responses of patients affected by UR at ${t}_{2}$, we choose to shift by − 1 all the difficulty parameters of the item(s) affected by recalibration, making all response categories easier. For patients affected by NUR, difficulty parameters were differentially shifted by values ranging 0 to 2 $2\eta ,\mathrm{ with~}\eta =1.8$: the first positive response category kept the same difficulty parameter over time, while other categories became more difficult. Finally, we kept the difficulty parameters constant over time to simulate the responses of patients not affected by RS.

$$\text{For all } m \,\mathrm{in }\left\{1,\dots ,M-1\right\},~ \delta_{jm}^{(t_2)} = \left\{ {\begin{array}{*{20}cl} \delta_{jm}^{(t_1)}+\eta_m & \text{for individuals affected by RS} \\ \delta_{jm}^{(t_1)} & \quad \,\text{for individuals not affected by RS} \end{array} } \right. $$

For UR, ${\eta }_{m}^{UR}=-1$ for all $m$ $\mathrm{in }\left\{1,\dots ,M-1\right\}$

For NUR, $\eta_{m}^{NUR}=\left\{\begin{array}{cll}\frac{(m-1) \eta}{m} & \quad\text { if } ~1 \leq m<\frac{M}{2}& \\ \eta & \quad\text { if } ~m=\frac{M}{2} & \text { where } \eta=1.8 \\ \frac{(M-m+1) \eta}{M-m} & \quad\text { if } ~\frac{M}{2}<m \leq M-1 & \end{array}\right.$