Better Information From Survey Data: Filtering Out State Dependence Using Eye-Tracking Data

Abstract

Ideally, survey respondents read and understand survey instructions, questions, and response scales, and provide answers that carefully reflect their beliefs, attitudes, or knowledge. However, respondents may also arrive at their responses using cues or heuristics that facilitate the production of a response, but diminish the targeted information content. We use eye-tracking data as covariates in a Bayesian switching-mixture model to identify different response behaviors at the item–respondent level. The model distinguishes response behaviors that are predominantly influenced either positively or negatively by the previous response, and responses that reflect respondents’ preexisting knowledge and experiences of interest. We find that controlling for multiple types of adaptive response behaviors allows for a more informative analysis of survey data and respondents.

Appendix

The appendix consists of five parts. First, we present the MCMC approach for estimating the multiple-response behavior (MRB) model. Second, we present evidence to support our selection of $M = 3$ classes in the application. Third, we illustrate how the MRB model infers the different response behaviors based on the observed responses. Fourth, we examine the marginal fit of the MRB model. Fifth, we present the results of a simulation study that demonstrate the empirical identification of the MRB model.

1.1 A.1 MCMC for the MRB Model

In this section, we develop the MCMC procedure to estimate the proposed MRB model for ordinal ratings. We assume independent prior distributions and conditional independence across subjects. To reduce clutter, we drop subscript i in the following expressions. The factorization of the joint probability function (4) suggests the following full Gibbs sampling scheme in which we cycle through conditionally independent distributions:

1.1.1 A.1.1 Sampling of the $z_i$

After dropping all constants with respect to z from the joint distribution:

$$\begin{aligned} p \left( z_{t} | y_{t}, c, \cdot \right) \propto p\left( y_{t} | z_{t},c \right) \times p\left( z_{t} | z_{t+1}, z_{t-1}, S_{t}, S_{t+1},\sigma _{+dep}^2,\sigma _{-dep}^2, \mu _{m,t},\sigma ^2_{m,t},m \right) . \end{aligned}$$

The detailed computations involve at most normal–normal updates and sampling from univariate truncated normal distributions.

In the course of our sampling scheme, we take advantage of the possibility to augment $z_{t}$ under all possible response behaviors. For example, the update of class-specific parameters that reflect different knowledge and experience sets behind non-contextual responses discussed below is facilitated by augmenting missing non-contextual responses for those classified as +dep, $-dep$, or observed to be DK.

Similarly, we augment missing responses and corresponding response behaviors to avoid an artificial constraint on response behaviors following a DK response. For example, the response vector $\{y_1, y_2, \mathrm{DK}, y_4, \mathrm{DK}, \mathrm{DK}, y_7, y_8, \ldots \}$ is represented as a complete vector $\{z_1, z_2, z_3, z_4, z_6, z_7, z_8, \ldots \}$ by temporarily excluding DK as a response behavior. To obtain z for DK responses, we treat the response behavior as missing and generate it from its hierarchical prior distribution renormalized to exclude DK. The draw of the corresponding z then proceeds, given the augmented response state, from an unconstrained normal distribution $p\left( z_{t} | z_{t+1}, z_{t-1}, S_{t}, S_{t+1},\sigma _{+{dep}}^2,\sigma _{-dep}^2, \mu _{m,t},\sigma ^2_{m,t},m \right)$.

This way, our model allows for the potential classification of $y_4$ or $y_7$ in the above example as contextual through the hierarchical prior despite the lack of direct likelihood information caused by DK responses for questions 3, 5, and 6. This is a useful feature of our model, because there is no theory that would guarantee non-contextual responses following a DK response. However, in the hierarchical regression that links probabilities over response behaviors to respondent- and item-specific covariates (Eq. 5), we only use “observed” response behaviors as dependent variables, i.e., DK in the case of a missing response.

1.1.2 A.1.2 Draw of Latent Response States

By Bayes’ Theorem, the full conditional distribution of $S_{t}=j$, conditional on the allocation of all other responses $S_{-t}$ and first-order dependence of the latent states, is given by:

$$\begin{aligned}&p \left( S_{t}=j|z,S_{-t},X, \beta , \cdot \right) = \frac{p\left( S_{t}=j,z_{t}|S_{-t},z_{-t},X, \beta ,\cdot \right) }{\sum _{j=1}^J p\left( S_{t}=j,z_{t}|S_{-t},z_{-t},X, \beta ,\cdot \right) }\nonumber \\&\quad =\frac{p\left( S_{t+1}|S_{t}=j,X_{t+1}, \beta \right) \times p\left( S_{t}=j|S_{t-1},X_t, \beta \right) \times p \left( z_t | S_t=j, z_{-t},\cdot \right) }{\sum _{j=1}^J p\left( S_{t+1}|S_{t}=j,X_{t+1}, \beta \right) \times p\left( S_{t}=j|S_{t-1},X_t, \beta \right) \times p \left( z_t | S_t=j, z_{-t},\cdot \right) }\nonumber \\ \end{aligned}$$

(A.1)

where $\cdot$ indicates parameters omitted for brevity. The above expression considers that changing $S_t$ changes the likelihood of “observing” $S_{t+1}$. In the case of $S_{t}=\lnot con$, this leads to:

$$\begin{aligned}&p \left( S_{t}=\lnot con|z,S_{-t},X,\beta , \cdot \right) \nonumber \\&\quad =\frac{p\left( S_{t+1}|S_{t}=\lnot con,X_{t+1}, \beta \right) \times p\left( S_{t}=\lnot con|S_{t-1},X_t, \beta \right) \times p \left( z_t | \mu _m, \Sigma _m \right) }{\sum _{j=1}^J p\left( S_{t+1}|S_{t}=j,X_{t+1}, \beta \right) \times p\left( S_{t}=j|S_{t-1},X_t, \beta \right) \times p \left( z_t | S_t=j, z_{-t},\cdot \right) }.\nonumber \\ \end{aligned}$$

(A.2)

The likelihood of observing $z_t$ is conditional on all other “non-contextual” latent evaluations, $z_t^{(\lnot con)}$ and class membership m. This is a univariate normal distribution where the conditional moments can be computed in the usual way. In the case of $S_{t}=\textit{contextual}$:

$$\begin{aligned}&p \left( S_{t}=+{dep}|z,S_{-t},X,\beta , \cdot \right) \nonumber \\&\quad =\frac{p\left( S_{t+1}|S_{t}=+{dep},X_{t+1}, \beta \right) \times p\left( S_{t}=+{dep}|S_{t-1},X_t, \beta \right) \times p \left( z_t | z_{t-1}, \sigma _{+{dep}}^2 \right) }{\sum _{j=1}^J p\left( S_{t+1}|S_{t}=j,X_{t+1}, \beta \right) \times p\left( S_{t}=j|S_{t-1},X_t, \beta \right) \times p \left( z_t | S_t=j, z_{-t},\cdot \right) }.\nonumber \\ \end{aligned}$$

(A.3)

And, finally, in the case of $S_{t}=-{dep}$:

$$\begin{aligned}&p \left( S_{t}=-{dep}|z,S_{-t},X,\beta , \cdot \right) \nonumber \\&\quad = \frac{p\left( S_{t+1}|S_{t}=-{dep},X_{t+1}, \beta \right) \times p\left( S_{t}=-{dep}|S_{t-1},X_t, \beta \right) \times p \left( z_t | z_{t-1}, \sigma _{-{dep}}^2 \right) }{\sum _{j=1}^J p\left( S_{t+1}|S_{t}=j,X_{t+1}, \beta \right) \times p\left( S_{t}=j|S_{t-1},X_t, \beta \right) \times p \left( z_t | S_t=j, z_{-t},\cdot \right) }.\nonumber \\ \end{aligned}$$

(A.4)

Draws of $S_t$ can be obtained directly from a multinomial distribution, given above expressions. However, we use independence MH sampling that proposes a new state from the hierarchical prior $p\left( S_{t}|S_{t-1},X_t, \beta \right)$, and accepts the move to the new state based on an MH ratio that only involves the ratio of the respective conditional likelihoods. While sampling directly from the multinomial distributions is numerically more efficient individually, we found this MH step to be advantageous in our application where classifications and (regression) parameters in the hierarchical prior over response behaviors need to converge jointly.

Note that we can write $p \left( S_{t}=\mathrm{DK}|z,S_{-t},X,\beta , \cdot \right)$ formally in the exact same way as above. However, as DK is observed or equivalently $p \left( z_t | S_t = \mathrm{DK} \right) = 0$ for $z_t$ corresponding to observed $y_t$, there is no need to update response behavior for DK, or allow for a positive posterior probability of DK when in fact a rating response is observed.

1.1.3 A.1.3 Draw of Parameters of Response Models

Update of $\{\mu _m\}, \{\Sigma _m\}$ A fully conjugate update of $\{\mu _m\}, \{\Sigma _m\}$, given allocation of responses to response behaviors, is achieved as follows. In case that a respondent’s set of responses contains DK, contextual ($+dep$) or contrasting ($-dep$) responses, we first augment these “missing” non-contextual responses $z^{(\lnot con)}_{miss}$, conditional on the “observed” non-contextual responses that determine a classification m. Because of conditional independence given a classification m into a latent class reflecting heterogeneous knowledge and experience underlying non-contextual responses we obtain:

$$\begin{aligned} p \left( z^{(\lnot con)}_{\textit{miss}} | \mu _m,\Sigma _m \right) = \prod _{t:miss} p \left( z_t | \mu _m,\Sigma _m \right) \end{aligned}$$

(A.5)

After having obtained a complete set of non-contextual z for all items, we can use fully conjugate results for the update of $\mu _m, \Sigma _m$ for each class independently.

Update of $\sigma _{+{dep}}^2, \sigma _{-{dep}}^2$ Given allocation to the states, we can collect all contextual responses across respondents and generate draws from the posterior distributions using standard results from the literature. The contrasting responses present a difficulty in that the likelihood does not have a standard format because of the exclusion of certain areas under the z-curve. We therefore use a RW Metropolis step for the draw of $\sigma _{-{dep}}^2$.

1.1.4 A.1.4 Draw of Latent Class Membership of Non-contextual Responses

Given allocation of responses to the states, the standard Bayes classifier is:

$$\begin{aligned}&p \left( m | z^{(\lnot con)}, \{\mu _m\}, \{ \Sigma _m\}, \eta \right) \nonumber \\&\quad =\frac{p\left( z^{(\lnot con)} | \mu _m, \Sigma _m\right) \times p\left( m|\eta \right) }{\sum _{m=1}^M p\left( z^{(\lnot con)} | \mu _m, \Sigma _m\right) \times p\left( m|\eta \right) } \end{aligned}$$

(A.6)

That is, we evaluate to observe the given $z^{(\lnot con)}$ under the various class and draw class membership given prior membership probabilities. The prior membership probabilities $\eta$ are updated as usual, given class assignments:

$$\begin{aligned} p \left( \eta | m, \cdot \right) \propto \prod _{i=1}^N p \left( m_i | \eta \right) \times p\left( \eta \right) \end{aligned}$$

(A.7)

which is a standard Dirichlet update, given our choice of a symmetric Dirichlet prior distribution for $\eta$:

$$\begin{aligned} p \left( \eta \right) = \textit{Dirichlet}\left( 3,...,3\right) \end{aligned}$$

1.1.5 A.1.5 Draw of $c_i$

Our model specifies individual-level cut-points to account for scale use heterogeneity. The posterior distribution of $c_i$:

$$\begin{aligned}&p \left( c_i| \cdot \right) \propto \prod _{t=1}^T p\left( y_{i,t}| c_i\right) \times p \left( c_i|\tau \right) \nonumber \\&\quad = \prod _{t=1}^T \left[ \int _{y_{i,t-1}}^{y_{i,t}}p \left( z_{i,t}| S_{i,t}, z_{i,t-1},\sigma _{+{dep}}^2, \sigma _{-{dep}}^2,\mu _m, \Sigma _m\right) \right] \times p \left( c_i|\tau \right) . \end{aligned}$$

(A.8)

For acceleration of convergence of the MCMC, we marginalize the likelihood of the data with respect to the $z_i$. That is, we compute the likelihood of observing the ratings $y_i$ integrated over the $z_i$ where $y_i$ for the purpose of this update contains the non-DK responses only. DK responses do not provide information for the cut-points which give rise to the ratings only. For the draw of the cut-points, we apply the procedure in Cowles (1996), using independent, non-informative uniform prior distributions for the cut-points.

1.1.6 A.1.6 Draw of $\beta ^{(reg)}$

Given draws of the latent states (when necessary), we recognize that the likelihood of observing (all) states is a multinomial distribution defined over the space of the response models. Using a standard prior distribution for $\beta ^{(reg)}$, we can update this parameter in the usual way, using a RW-MH update (Rossi et al. 2005).

1.2 A.2 Robustness of Results with Respect to Number of Latent Classes

In this part of the appendix, we explore robustness of our results with respect to the number of latent classes of non-contextual responses. The results presented in Sect. 4 are based on $M=3$, identified as the best performing model in terms of predictive fit. We note, however, that, for $M \in \{1:8\}$, the out-of-sample fit of our model does not change much and declines markedly for $M \ge 9$. In other words, the empirical signal from our data in favor of a particular solution in $M \in \{1:8\}$ is not very strong. However, larger M (more persistent, non-contextual heterogeneity a priori) allows for a more flexible modeling of (any) response pattern which raises the question to what extent inference regarding different response behaviors (“contextual” [+dep], “contrasting” [$-dep$], and “non-contextual”) depend on M.

1.2.1 A.2.1 Allocation to Latent States

We start by looking at the shares of responses states, by item, for different M (see Fig. 8). With respect to non-contextual responses (uppermost panel), we find a general tendency for their shares to increase as M increases. This is to be expected given the implied larger flexibility of a model of non-contextual responses with more latent classes. However, we also find the impact of a change in M to be rather small. Given the range of M considered here, the (across-item) mean of the standard deviation of shares of non-contextual responses (across M-levels) is only 0.026. With regard to contextual ($+dep$) responses (lowermost panel): we find that an increase in M to decrease their shares. Again, these changes in shares are small implying that allocation is robust with respect to M. For contrasting ($-dep$) responses (middle panel), this is partially different. In particular for item 3 (“Facilities”), we find that the share of negatively dependent responses changes significantly. We observe the lowest share of this state for $M=8$ ($5.7\%$) and the highest share for $M=6$ ($20\%$). Similar notable changes can be observed for items 14 (“Connect national transportation system”) and 20 (“City size”) (Fig. 8). The shares of the other 24 items, however, exhibit only minor changes.

Fig. 8

Posterior distribution of response behaviors by item for $M \in \{1:12\}$. Legend: Shares shown are based on posterior means of allocation results. Solid lines indicate shares for $M=1,3,12$. For better readability shares, for $M\in \{2,4:11\}$ are all in gray dotted lines

Related to variability of allocation results as a function of M is the question whether non-contextual responses exhibit different response patterns when M is changed. Eq. (3) specifies non-contextual responses as a mixture distribution where each response is obtained by marginalizing over latent classes. Figure 9 shows the means of non-contextual responses after marginalizing over latent classes, given different M. The question to answer here is whether non-contextual response patterns across items differ after integrating out persistent heterogeneity (integrating over LCs). Ideally, they would not. Figure 9 reveals that the number of LC has very little influence on non-contextual responses. This result is consistent with the small effect of M on response behavior allocation (see Fig. 8).

Fig. 9

Means of non-contextual responses, marginalized with respect to latent classes, given $M \in \{1:12\}$. Legend: Shares shown are based on posterior means for each item. Solid lines indicate shares for $M=1,12$. Gray dotted lines indicate shares for $M\in \{2:11\}$

1.2.2 A.2.2 Conditional Variances of First-Order States

Figure 10 shows the posterior means of the variance of the first-order dependency states for different M. From Fig. 10, it is clear that contextual ($+dep$) response behavior is unaffected by changes in M (within the range considered here). The conditional variance indexing contrasting ($-dep$) response behavior exhibits a slight but noticeable and non-monotonic decline as M increases. This is consistent with the less clear cut identification of this response behavior based on eye-tracking covariates.

Fig. 10

Posterior means of variances of first-order states for $M \in \{1:12\}$

1.2.3 A.2.3 Counterfactual Analysis

Figure 11 shows the differences between non-contextual responses and all responses on the Z-level, given different numbers of latent classes. Figure 11 reveals that the differences between all responses and responses classified as non-contextual do not depend on the number of latent classes used to model persistent heterogeneity. This analysis also confirms the items that are, marginally, most affected by contextual response behavior.

Fig. 11

Differences between non-contextual responses and all responses given $M \in \{1:12\}$. Displayed are differences on Z level. Legend: Differences for $M=1$ and $M=12$ are displayed in solid red and green lines, respectively. Solutions for $M \in \{2:11\}$ are displayed in dotted gray lines

1.3 A.3 Illustrative Example of Response Allocation Using the MRB Model

To demonstrate the predictions of the MRB model, we show in Fig. 12, by way of example, the latent evaluations (i.e., on the z level) of respondent 405 in our survey. The graph in Fig. 12 also contains box plots of the non-contextual responses by item, given this respondent’s allocation to the latent non-contextual classes. Figure 12 provides additional insights into how our model works. Responses to items which fall into the 75% range of the non-contextual latent class for an item are more likely to be allocated to the non-contextual response behavior (e.g., items 1, 2, 5, 11, 15). Evaluations on a level inside the whiskers, but outside the 75% range of the latent class, are allocated to the non-contextual class, given large 1st differences (e.g., items 24, 27). Evaluations exhibiting small 1st differences and outside the 75% range of the latent class are likely to be classified as “contextual” (e.g., items 3, 6, 21). Evaluations exhibiting large 1st differences and outside the 95% range of the non-contextual class are more likely to be allocated to the “contrasting” response behavior (item 25). The plot also shows exceptions to these observations. For example, item 23 is categorized as “non-contextual”, although the latent z lies outside the 95% range of its non-contextual class and the observed response is a perfect copy of the previous rating. In this case, information from eye traces drives the allocation, suggesting that the information processing pattern for this item is consistent with a non-contextual response and not copying to reduce effort.

Fig. 12

Latent evaluations and response allocation. Legend: The graph shows, as circles, the latent Z for respondent 405 and the allocation of each evaluation to the response behaviors by posterior mode (states are color coded). DK responses are given an arbitrary value (0) for identification. The latent Z of this respondent are laid over box plots of the latent non-contextual Z for this respondent’s non-contextual state. The horizontal dotted lines indicate the cut-points for the respondent so that the ratings of each item, given the z, can be inferred (except for DK). Note that a rating to the 1st item in the survey is always allocated to the “non-contextual” group because it cannot exhibit first-order dependency

Fig. 13

Posterior predicted rating frequencies in [1 : 3]

Fig. 14

Posterior predicted rating frequencies in [4 : 6]

Fig. 15

Posterior predicted rating frequencies in [7 : 9]

Fig. 16

Posterior predicted DK

This example demonstrates why allocation of ratings to response behaviors should be model-based and not deterministic or based on pre-defined rules. For example, we find that responses which exhibit small first differences are not necessarily contextual and that 1st differences larger than zero do not exclude the contextual response mode. Compare, e.g., items 1 and 2 for this particular respondent. The rating to item 2 is a copy of the previous rating. However, given the distribution of non-contextual responses to item 2 in this class of non-contextual responses, a copied rating is a likely response, given the non-contextual state. The same observation holds for items 7 and 8. Another useful example is presented by item pair 12–13. Our model classifies the response to item 13 as “contextual” despite a 2-point first-order difference. A similar case is presented by item 26. It appears that responding contextually cannot be equated with copying responses. Our example in Fig. 12 also shows why the contrasting response model is a useful addition to the MRB model. The response to item 25 exhibits a 1st difference of 7 rating scale points. This very large difference implies that a contextual response is highly unlikely. Given the distribution of non-contextual responses to item 25, it is also not very likely to be non-contextual. Thus, relative to the other response behaviors, allocation to the contrasting state is preferable.

Our model accounts fully for uncertainty with respect to allocating responses to latent response behaviors. For example, the response of respondent 405 to item 14 Fig. 12 has a 57.2% probability to be allocated to the contrasting state, giving rise to the posterior mode of the allocation as “contrast”. However, the probability of being allocated to the non-contextual state is 40.0%, indicating that a response 3 rating scale points away from the expected non-contextual rating does not push this probability to zero. The results from the hierarchical regression presented in the previous section account fully for such uncertainty in allocation. Note that the hierarchical regression also includes the effect of covariates on the observed DK state which is observed a priori and, thus, not subject to allocation uncertainty.

1.4 A.4 Model Fit Assessment

To assess the “absolute fit” of the selected model, we simulated posterior predictive data sets conditional on the calibration data and the model estimates. These simulation studies showed that the selected model fits the data well with one exception. The following plots visualize our results and show how the posterior predictions track the observed variation across items in the rating ranges 1:3, 4:6, and 7:9 (see Figs. 13, 14, and 15). The red lines in the plots trace out the observed frequencies of responses in a rating range across the 27 items. The box plots summarize realizations from different posterior predictive data simulations. We see that data generated from the best-fitting model track the patterns in the data well except for item 24 “Support by city administration” in the rating range 4:6 (Fig. 14). This item is special in that 43% of observed responses to this item are DK (see Table 5). Recall that our model predicts DK essentially only based on eye-tracking covariates which is not sufficient for this item. Figure 3 also shows that the hierarchical prior over response behaviors is much less informative for item 24 than for other items. Hence, predictions for this items suffer (see also Fig. 16 that compares model-based predictions of DK against observed DK across items).

Table 9 Parameters of Data Simulation and Recovery via MCMC

Fig. 17

Recovery of Parameters of Hierarchical Regression

Fig. 18

Recovery of Parameters of non-contextual Latent Classes. Means (upper panel) and variances (lower panel) of non-contextual responses

1.5 A.5 Simulation Study

In this section, we demonstrate empirical identification of the proposed MRB model. For this purpose, we simulate data from the model and demonstrate recovery of the parameters of the model, using the MCMC procedure outlined in the appendix to this paper. For our simulation study, we generate data from 400 respondents and 25 items, a setup similar to the data set used in the empirical analysis. We specify a hierarchical regression to introduce prior information to the latent states and generate observed ratings on a 9-point scale via cut-points. Table 8 outlines the model parameters (with the exception of the parameters of the non-contextual states and the hierarchical regression to reduce clutter) and recovery via our MCMC. For the parameters of the non-contextual state and the upper level model, we present true and recovered parameters (posterior means) in Figs. 17 and 18. All results reported indicate that the MCMC recovers the true parameters at reasonable levels of accuracy. Note that small variances of latent responses z can, theoretically, only be recovered up to a level indicated by the number of rating scale points.

We note that, in general, empirical identification of the proposed MRB model hinges upon informative likelihoods of the latent states, given model parameters, and/or sufficient prior information regarding the latent response styles. The setup in the simulation study reported here presents a favorable case in that both the lower level of the model (response behaviors) and the upper level (hierarchical regression) are informative. It is, of course, possible to generate non-contextual responses in a way so that these are indistinguishable from, e.g., contextual responses. This is the case when, for a set of two subsequent non-contextual responses, the means are similar and the variances of the non-contextual responses and the residual variance of contextual responses are small. Then, likelihood information to distinguish the two states for the second item in the set is weak. A similar situation arises when first-order mean differences between non-contextual responses suggest contrasting behavior at a similar likelihood level. In these types of situations, prior information, e.g., via eye tracking, becomes important to effectively distinguish response styles. However, through additional simulations not reported here, we found that it is possible to recover model parameters when neither covariates nor the “observed” responses are sufficient by themselves for identification so that, to recover the true latent response styles, both observables are necessary. However, distinguishing the latent response strategies becomes difficult when both likelihood level and upper level information are weak.