1 Introduction

Contract survey research organizations such as Amazon Mechanical Turk (MTurk), have become an efficient and popular means of recruiting online survey study participants. These contract survey research organizations often have millions of potential survey respondents willing to complete surveys for a modest remunerative fee [12, 19]. However, there are inherent risks to recruiting survey participants through such organizations. Given that remuneration is often based on survey completion regardless of the veracity of their responses, some participants may be tempted to misreport their responses in order to complete the survey more expeditiously [3, 16].

Misreporting in health surveys is a critical issue that can seriously threaten the validity and credibility of public health research. Misreported data can lead to counterproductive and/or ineffective public health policy decisions and misallocation of resources [5, 18]. Implementation of such ineffective public health policies in turn has the potential to undermine public trust in not only researchers and specific policies, but public health institutions more broadly.

In China, Sojump (also known as wenjuanxing) is a popular source for recruiting survey participants [10, 21]. Sojump claims that their pool of survey respondents number more than 10 million individuals living in China [21]. However, to the best of our knowledge, no study has yet examined the correlates or ubiquity of online survey misreporting in China.

1.1 “Watching eyes” and deviant behavior

The “watching eyes” effect is a phenomenon in which viewing images or representations of eyes cause people to change their behavior, such as refraining from deviant behavior [6, 11]. This inverse relation between exposure to “watching eyes” and deviant behavior is hypothesized to be driven by evolutionary biological mechanisms. Researchers theorize that humans developed a strong sensitivity to being watched and regulated behavior to defend ourselves from predators and to maintain our reputation in social groupings [4, 9, 15, 17, 20, 22]. The theorized rationale of the “watching eyes” effect can be divided into two aspects, namely, reputation management and standardization [1]. Individuals make more prosocial choices and exhibit more altruistic and generous behaviors in the presence of eye images [1]. Many previous studies have shown that individuals will strategically change their behavior when they are observed by them, because people probably tend to manage their reputation in front of other people [4]. Standardization refers to the behavioral standard that people need to abide in a certain social group. “Watching eyes” image could create a sense of being watched and drive people to social conformity [1, 4].

Hence, “watching eyes” images have been deployed in various settings to deter deviant behavior. For example, in the UK, “watching eyes” have been widely applied to the rail network to reduce crime [11].

1.2 Lacuna on watching eyes & internet survey misreporting

Experimental research from China found that exposure to geometric designs resembling mammalian eyes had no association with deviant behavior [7]. However, one possible explanation for this null finding was that visual designs RESEMBLING human eyes are insufficient for stimulating consciousness of being watched. It is possible that photos of actual human eyes will elicit a stronger activation of consciousness of being watched, compared to eye-like visual designs.

In any case, to the best of our knowledge, no study in China has specifically examined correlates of online survey misreporting, and no published study anywhere has examined the effect of eye-like or actual human “watching eyes” on survey misreporting.

1.3 Study objectives and hypothesis

In response to these concerns, we conducted an online experiment in China that: (1) examined the ubiquity and sociodemographic correlates of survey misreporting, and (2) examined the effects of “watching eyes” on survey misreporting. We hypothesized that exposure to watching eyes would be associated with less misreporting.

2 Materials and methods

2.1 Study design

This study was conducted among Chinese adults aged 18 years old and above via an online survey of the general population in Mainland China from August to October 2022. Survey participants were recruited through Sojump’s contract survey research services, which is based upon their pool of approximately 10 million uniquely registered users [21]. The study was approved by the research ethics committee of Xi’an Jiaotong-Liverpool University, and conducted in accordance with the principles of the Helsinki Declaration. The researchers were provided with only the anonymous database after the survey data were collected from the SoJump survey service. No identifiable information was collected, and no individual participant could be identified. The questionnaire is available upon request to the corresponding author.

2.2 Measures

The following respondent characteristics were measured through self-report in the survey: gender, age, education level, current city residence, marital status, occupation, monthly salary, and blood type.

2.3 Experimental design

Eligible study participants were randomized to one of four arms using the survey platform’s randomization function (Fig. 1). Participants randomized to arms 1 and 3 were asked to simply report their blood type with the following question, “What is your blood type? AB, A, B, O, or I don’t know”. Participants randomized to arms 2 and 4 were also asked to report their blood type, but were additionally shown the following text: “If you have type AB blood, then you can end the survey early; otherwise, you need to continue answering additional questions.” In short, individuals with non-AB type blood had an incentive to misreport their blood type as type AB blood if they were randomized to either arm 2 or 4; however, individuals with non-AB type blood had no such incentive if they were randomized to arm 1 or 3.

Fig. 1
figure 1

Study arms

Full size image

Reporting of type AB blood was used as the indicator of misreporting for two key reasons. (1) Individual study participants likely knew that researchers could not verify their individual blood type and thus participants would not have felt inhibited by concerns about being “caught” misreporting their blood type. (2) Although we were unable to confirm the veracity of any individual’s self-reported blood type, national public health surveillance data in China suggests that approximately 1/10 of any study arm in the aggregate would truly have type AB blood [23]. Therefore, inferences about misreporting could made at the group-level, as has been done in previous experiments examining intervention effects on socially deviant behaviour [24].

Participants randomized to either arm 3 or 4 were shown a graphic of watching eyes before being asked to report their blood type (Fig. 2). The graphic was shown for a minimum of three seconds, after which the participant could advance to the question concerning blood type. Participants randomized to arm 1 or 2 were not shown the watching eyes graphic.

Fig. 2
figure 2

Watching eyes graphic

Full size image

2.4 Statistical analysis

To assess success of randomization, descriptive statistics were firstly shown for all participants and then among participants stratified by study arms. We used Chi-squared tests to: (1) identify sociodemographic correlates of self-reported AB blood and (2) assess whether the frequencies of reporting AB blood type significantly differed by study arm. Data were analyzed in SPSS (Chicago, IL, USA). P-values < 0.05, two tailed, were considered statistically significant.

3 Results

Participants were recruited in two stages. In stage one, 1204 individuals were screened for eligibility in August 2022, 19 of whom were excluded because they did not live in Mainland China or were not at least 18 years old. Thus, in stage one 1185 participants were recruited. Due to the low numbers of individuals with lower educational attainment in stage one, we conducted a second stage of recruitment that only included individuals with less than college level education. In stage two, 521 individuals with high school or lower educational attainment were screened for eligibility in October 2022, 51 of whom were excluded because they did not live in Mainland China or were not at least 18 years old. Thus, in stage two 470 participants were recruited. The combined overall sample size was 1655.

3.1 Participant characteristics

Stage one participant characteristics stratified by study arm are presented in Table 1. Most of the respondents were likely to be female (57.38%), married (62.28%), below 40 years old (88.95%), with an education level of college or above (91.73%). 14.43% of stage one participants reported AB blood type.

Table 1 Characteristics of stage-one participants (N = 1185)
Full size table

Stage two participant characteristics are presented in Table 2. Most of the respondents were likely to be female (51.28%), married (60.21%), and below under 40 years old (76.17%). Approximately 15.32% of stage two participants reported AB blood type.

Table 2 Characteristics of stage-two participants with high school or lower educational attainment (N = 470)
Full size table

3.2 Correlates of reporting AB blood type

Table 3 shows the characteristics of participants (stages 1 and 2 combined) reporting AB blood type, stratified by subgroup and study arms 1 and 2. No studied characteristics were found to be significantly proportionally different between these two arms (No incentive to report AB blood, no watching eyes vs. Incentive to report AB blood, no watching eyes).

Table 3 Percentage reporting AB blood type, by study Arm and subgroup, among individuals recruited from stages 1 and 2 combined
Full size table

3.3 Effect of “watching eyes” intervention on reporting AB blood type

Among the stage one participants who were not given an incentive to report AB blood type, there was no significant difference in the probability of reporting AB blood type between those who were not and were shown watching eyes (arm 1: 12.17% vs arm 3: 14.09%, p = 0.54).

However, among stage one participants who were given an incentive to report AB blood type, the probability of reporting AB blood type was significantly greater among individuals who were shown watching eyes (arm 4: 19.93% vs arm 2: 11.56%, p < 0.01).

4 Discussion

Contract survey research organizations have become a popular source of participant recruitment in China [21], but there has been virtually no research that examines tendencies for misreporting, let alone interventions to discourage deliberate survey misreporting. In the present study, we sought to identify sociodemographic correlates of survey misreporting at the group-level, and examined the effects of watching eyes visual graphics as a deterrent to survey misreporting.

Experimental research has suggested that dishonest behavior is correlated with sociodemographic characteristics [13]. Although we observed that individuals with the lowest level of educational were over twice as likely to report type AB blood when there was incentive to do so, this difference did not reach statistical significance. Larger studies are needed to assess if this lack of statistical significance was due to the relatively small number of participants who had less than high school educational attainment.

Aside from educational attainment, we did not detect evidence that any other sociodemographic subgroup was more or less likely to misreport their blood type when there was an incentive to do so. One possible explanation is that the incentive to end the survey early was not sufficiently enticing. All study participants were assured a monetary reimbursement from the contract survey company upon completion of the survey, regardless of whether they completed the survey early or not. However, if the incentive to report AB blood type had been of greater potential value (e.g., raffle prize drawing only available to individuals who were AB blood type), then more individuals may have deliberately misreported their blood type as AB, as suggested by previous research [8].

Contrary to our original expectations, results indicated that the watching eyes graphic actually significantly increased reporting of AB blood type when there was an incentive to do so. This result contrasts with previous research from China which found no effect of watching eyes on socially deviant behavior, and may be because of differences in eye cue visuals [7]. The current study used a photo of actual human eyes which may have had a stronger activation of consciousness of being watched, whereas the study by Cai et al. only used a visual graphic of geometric shapes and lines that vaguely resembled mammalian eyes (2015).

We speculate that participants exposed to the watching eyes may have experienced greater unease due to a sense of being watched, and therefore deliberately misreported their blood type as AB in order to end the survey sooner. This is partly corroborated by the observation that participants exposed to watching eyes were NOT significantly more likely to report AB blood type when there was no incentive to do so. Future studies and interventions should be mindful that watching eyes could potentially induce unwanted feelings of being watched that ultimately cause individuals to completely disengage from the activity. For example, if watching eyes are deployed on an online survey that is done voluntarily without any remuneration linked to completion of the survey, then it is possible that some eligible and truthful individuals will see the watching eyes and drop out of the survey prematurely due to unwanted feelings of being watched. Combining watching eyes with explicit social norm setting around the behavior of interest may help participants understand the purpose of the watching eyes [2, 14], be more tolerant of the feelings of being watched, and less likely to misreport.

4.1 Limitations

Several study limitations should be noted. First, the incentive we applied in this online survey may have been relatively weak. If the incentive to misreport had been greater (e.g., monetary reward), then we may have observed significantly greater levels of reporting AB blood type when there was an incentive to do so. Second, the limited number of individuals with less than high school education may have constrained power to detect statistically significant effects among this subgroup. Third, individuals’ true blood type could not be ascertained, so inferences about misreporting could only be made at the group level, as has been done in previous experimental research [24].

5 Conclusion

This online experimental study is among the first in China to examine correlates of survey misreporting and the effects of watching eyes on internet survey misreporting. Findings indicate that individuals with lower educational attainment may be more likely to misreport when presented with an incentive, but additional research is needed to corroborate this result. Researchers and practitioners who use internet health surveys in China should begin rigorously assessing the correlates and pervasiveness of survey misreporting.

Study results also suggest that watching eyes interventions may have actually caused greater deliberate misreporting, potentially because exposure to watching eyes may have induced negative feelings that undermined willingness to cooperate. Policy makers and practitioners should be mindful that exposure to images of watching eyes may potentially discourage engagement and cooperation with surveys or interventions.