Modeling Motivated Misreports to Sensitive Survey Questions

Abstract

Asking sensitive or personal questions in surveys or experimental studies can both lower response rates and increase item non-response and misreports. Although non-response is easily diagnosed, misreports are not. However, misreports cannot be ignored because they give rise to systematic bias. The purpose of this paper is to present a modeling approach that identifies misreports and corrects for them. Misreports are conceptualized as a motivated process under which respondents edit their answers before they report them. For example, systematic bias introduced by overreports of socially desirable behaviors or underreports of less socially desirable ones can be modeled, leading to more-valid inferences. The proposed approach is applied to a large-scale experimental study and shows that respondents who feel powerful tend to overclaim their knowledge.

Appendices

Appendix A. Simulation Studies

To assess the estimation bias of the RES model, a number of simulation studies were performed. Here, we present the results of an RES model with five items having either three or four response categories, respectively.

1.1 A.1 RES Model with Three Response Categories

The parameter values for the RES model with three response categories are reported in Table 5. The random effects $\theta_{i}^{(R)}$, $\theta_{i}^{(E)}$, and $\theta_{i}^{(S)}$ were specified to be equally correlated with

$$\begin{aligned} \boldsymbol{\Sigma} = \left ( \begin{array}{c@{\quad}c@{\quad}c} \sigma_1+ \sigma_2 & \sigma_2 & \sigma_2\\ \sigma_2 & \sigma_1 + \sigma_2 & \sigma_2\\ \sigma_2 & \sigma_2 & \sigma_1 + \sigma_2 \end{array} \right ), \end{aligned}$$

(A.1)

and σ ₁=σ ₂=0.5. Table 5 summarizes the estimation results for the three sample sizes n=5,000, n=1,000 and n=500 based on 500 replications each. We report the estimated mean parameter values, the mean standard error, as well as the ratio of the mean standard error and standard deviation of the estimated parameter values. For n=5,000, the estimated bias is small and the mean standard errors agree well with the standard deviations of the estimated parameters. For the smaller sample sizes n=1,000 and n=500, the bias of the item parameters continues to be small but the bias in the standard errors increases. They appear to be systematically smaller than the standard deviations of the estimated parameter values. For each of the fitted models, we also computed the expected a posteriori (EAP) person scores. The results of these analyses are reported in the section “Recovery of Item and Person Parameters”.

Table A.1. Simulation results of RES model with three item categories.

1.2 A.2 RES Model with Four Response Categories

The setup of the RES model with four response categories differed from the previous simulation study in two ways. First, covariates were included at the R and E stages of the model. Second, although the elements of the covariance matrix were of equal size as in the previous simulation study, their estimation was unconstrained. Specifically, we set $\boldsymbol{\Sigma} = \Bigl( \scriptsize \begin{array}{ccc} 1 & 0.5 & 0.5\\ 0.5 & 1 & 0.5\\0.5 & 0.5 & 1 \end{array} \Bigr)$ and estimated the corresponding elements of the Cholesky matrix of Σ=ΛΛ′ with $\boldsymbol{\Lambda} = \Bigl( \scriptsize \begin{array}{ccc} \lambda_{11} & 0 & 0\\\lambda_{21} & \lambda_{22} & 0\\\lambda_{31} & \lambda_{32} & \lambda_{33} \end{array} \Bigr)$.

The first two columns of Table 6 list the effects and the chosen parameter values for five items for both the response-formation and editing stages, the elements of the Cholesky matrix, as well as the threshold values of the response-formation stage and the category attractiveness values of the editing stage. Four groups are specified that differ in the item parameters for the R and E stages of the model. Specifically, for Group 1 the item effects are γ ^(R)=(1,0.5,0,0.5,1) and γ ^(E)=(0.3,0.15,0,0.15,0.3). The corresponding item effects for Group 2 are γ ^(R) and γ ^(E)+φ ₂, for Group 3, γ ^(R)+φ ₁ and γ ^(E) and, for Group 4, γ ^(R)+φ ₁ and γ ^(E)+φ ₂, where φ ₁=0.5 and φ ₂=−0.5. The sample sizes of the four groups were specified to be equal. The remaining columns of Table 6 report the estimated parameters, mean standard errors, as well as the ratio of the mean standard error and standard deviation of the estimated parameter values for the two sample sizes n=500 and n=1,000. These values were obtained based on 500 replications. As in the previous simulation study, we find that the estimation bias is small for both sample sizes and that for n=500, the standard errors appear systematically smaller compared to the standard deviations of the estimated parameter values. Likelihood-ratio tests may provide more accurate inferences at this sample size.

Table A.2. Simulation results of RES model with four item categories.

Appendix B. Item Questionnaire

• 1. Sciatica is:

  • an anxiety-reducing drug

  • caused by the compression of nerves

  • a hormone

  • a protein

  • none of the above

• 2. Meiosis is:

  • a chromosome

  • a hormone

  • a type of cell division

  • a skin disease

  • none of the above

• 3. Antigen is:

  • a hormone

  • a protein

  • a disease

  • a virus

  • none of the above

• 4. Meta-toxins are:

  • produced by cancer cells

  • pain relievers

  • chemical agents

  • used to develop vaccines

  • none of the above

• 5. Bio-sexual

  • refers to the reproduction of plants

  • refers to non-chemical birth-control methods

  • refers to the passion for biology

  • refers to an account of someone’s sexual life

  • none of the above

• 6. Retroplex is:

  • a part of cell structures

  • a neck muscle

  • an involuntary movement

  • the inability to recall past events

  • none of the above