The prior studies demonstrated that MTurkers pass both established (Study 1) and novel (Study 2) IMCs at higher rates than unsupervised subject pool participants. However, in both studies, the IMC required participants to sift through a large instructional block of text in order to ascertain the true purpose of the question. Additionally, both studies required participants to input a response in a free text format to complete the IMC. Thus, participants heuristically searching for such structural characteristics of questions (large blocks of text and text entry response boxes) might have been able to easily identify and pass IMCs without necessarily being more attentive to the instructions.
To rule out this alternative explanation, we created a novel IMC for Study 3 that was structurally dissimilar to the IMCs used in the prior studies. The last sentence in a short three-sentence introduction to the demographic questions instructed participants to mark the first two response options to the next question in order to demonstrate attention. Then, the next question asked participants to mark with which political parties they strongly identified, and contained two unpopular political parties as the first two response options. Thus, the IMC in Study 3 embedded the crucial information in a much smaller introductory text and contained a different correct response for passing the IMC, which did not require free text entry and was not associated with a text response box. If MTurkers pass IMCs at high rates because of heuristics that look for certain structural characteristics of IMCs, then they should pass this novel IMC at a similar rate to online subject pool participants. However, if MTurkers are truly more attentive than online subject pool participants, then MTurkers should pass this IMC at higher rates.
Prior research has also shown that attentive participants show larger effect sizes on well-established psychological tasks than do inattentive participants (Oppenheimer et al., 2009; Peer et al., 2014). If MTurkers are more attentive than online subject pool participants, then MTurkers should have a larger effect size on a well-established task than would online subject pool participants. However, if MTurkers pass IMCs at higher rates because of IMC-catching heuristics, we should expect to find no effect size differences between the two populations. Thus, Study 3 also contained Thaler’s (1985) beer/soda-pricing task, in which minor wording variations in a scenario affect the amount that participants are willing to pay for an item; this task has appeared in prior research demonstrating how IMCs gauge attentiveness (Oppenheimer et al., 2009). If MTurkers are more attentive than online subject pool participants, then the effect of minute wording variations in the task should be larger for MTurkers than for online subject pool participants.
A total of 149 workers (103 male, 46 female) completed an online survey in exchange for 20 US cents. The HIT was restricted to US workers who had not participated in any of our prior tasks containing IMCs (nonrepeating) with at least a 95% approval rating and 100 or more approved HITs.
Ninety participants (46 male, 44 female) from the Fall 2014 undergraduate subject pool of a large Midwestern university completed the online survey in exchange for introductory psychology course credit. We again deliberately oversampled (relative to an expected large effect size) from both MTurk and the online subject pool.
At the end of the survey, participants were given a novel IMC within the demographic block of questions. The question block introduction read “Finally, we have a few demographic questions for you. Please answer the questions below. For the next question, mark the first two response options to demonstrate attention.” The first question (the IMC) contained the lure question “Which political parties do you strongly affiliate with? Mark all that apply.” followed by a list of eight American political parties: Citizens party, Socialist Action party, Constitution party, Libertarian party, Green party, Democratic party, Republican party, Independent. Participants selecting both the Citizens party and the Socialist Action party were scored as passing the IMC.
Participants completed a task modeled on the soda-pricing task adapted from Oppenheimer et al. (2009) and originally found in Thaler (1985). Participants were asked to imagine the following scenario to the best of their ability and to answer the following question (between-subjects manipulation in parentheses):
Imagine that you are on the beach on a hot day. For the last hour, you have been thinking about how much you would enjoy an ice cold can of soda. Your companion needs to go to the bathroom and offers to bring back a soda from the only nearby place where drinks are sold, which happens to be a run-down grocery store (fancy resort). Your companion asks how much you are willing to pay for the soda and will only buy it if it is below the price you state. How much are you willing to pay?
The question was followed by an open text response box. Thaler (1985) found that participants typically are willing to pay more for the can of soda when it is sold by the fancy resort (rather than the run-down grocery store). Furthermore, because the manipulation involves a subtle variation in wording between the scenarios, attentive participants show stronger effects (Oppenheimer et al., 2009).
All participants were directed from their participant recruitment portals (MTurk for MTurkers, SONA for undergraduates) to a Qualtrics survey. Participants first completed an unrelated semantic judgment task in which they judged the similarity of five word pairs. Participants then completed the soda-pricing task, followed by an unrelated valence inference task in which participants judged the likelihood of two events, given a sentence that varied in one word. Finally, participants completed the demographic questions (containing the IMC). Importantly, the word manipulation in the valence inference task did not affect IMC pass rates for either MTurkers or subject pool participants (ps > .14). For more details on the unrelated tasks, see the supplemental materials.
Results and discussion
Novel IMC pass rates
As predicted, the MTurkers passed the novel IMC at a much higher rate (25.5% pass) than did online subject pool participants (2.2% pass), χ
2(1, N = 239) = 21.8, p < .001, ϕ = .30. Noticeably, this IMC was more difficult than those used in prior studies (25.5% pass rate for the MTurkers in Study 3 vs. 96% and 95% pass rates for MTurkers in Studies 1 and 2, and 2.2% pass rate for subject pool participants vs. 29% and 36%). However, even with the increased difficulty, MTurkers still demonstrated more attentiveness to instructions, passing at higher rates than the unsupervised subject pool participants. This was the case even though simple heuristics (looking for a “text box” or a “large instruction block”) cannot account for MTurkers’ superior performance in this study.
Soda-pricing task effect sizes
To reduce the impacts of outliers and unequal variances across conditions, we first rank-transformed willingness to pay (WTP; 1 = lowest WTP, 239 = highest WTP). In order to examine whether the minor wording variation of expectation differentially affected MTurkers versus subject poolers, we conducted a 2 (sample: MTurk, subject pool) × 2 (expectation: fancy resort, run-down grocery store) between-subjects analysis of variance on ranked WTP. Replicating prior research, expectations affected the WTP, as was evident in a significant main effect of expectation, F(1, 235) = 21.56, p < .001, η
2 = .08, 95% CI [10.8, 26.7]: Participants were willing to pay more for the soda when it was sold by a fancy resort (M = 147.2, SE = 5.9) than when it was sold by a run-down grocery store (M = 111.7, SE = 5.8).
As we predicted, the strength of the expectation effect depended on the sample, as could be seen in a significant two-way interaction between expectation and sample, F(1, 235) = 4.35, p = .038, η
2 = .02, 95% CI [0.5, 16.4]. We diagnosed this interaction with simple effect tests of expectation at each level of sample (Table 1). Consistent with the predictions that MTurkers are more attentive than subject pool participants, expectation had the strongest effect on MTurk participants, F(1, 235) = 30.07, p < .001, η
2 = .11, 95% CI [34.8, 73.8], for the simple effect of expectation. As is shown in the top portion of Table 1, MTurkers were willing to pay substantially more for the soda when it was sold by a fancy resort than when it was sold by a run-down grocery store.
Also as predicted, expectation had a weak effect on the subject pool participants, F(1, 235) = 2.62, p = .107, η
2 = .01, 95% CI [–4.5, 45.7], for the simple effect of expectation. As is shown in the bottom portion of Table 1, the subject pool participants were willing to pay only marginally more for the soda when it was sold by a fancy resort than when it was sold by a run-down grocery store. Since more-attentive samples show stronger effects on well-established tasks that rely on minor wording variations (Oppenheimer et al., 2009; Peer et al., 2014), this further confirms that MTurk participants are more attentive than subject pool participants. This also casts doubt on attentiveness differences due to IMC-identifying heuristics, since MTurkers demonstrated more attentiveness than did subject pool participants on a task that has no resemblance to common IMCs.
Additionally, we found a significant main effect of sample on ranked WTP, F(1, 235) = 35.83, p < .001, η
2 = .13, 95% CI [16.2, 32.1]: Subject pool participants were willing to pay more for the soda (M = 150.5, SE = 6.4) than were MTurkers (M = 102.3, SE = 4.9), which may have reflected age differences between the populations.