Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk

Original Paper

Abstract

Online experiments allow researchers to collect datasets at times not typical of laboratory studies. We recruit 2336 participants from Amazon Mechanical Turk to examine if participant characteristics and behaviors differ depending on whether the experiment is conducted during the day versus night, and on weekdays versus weekends. Participants make incentivized decisions involving prosociality, punishment, and discounting, and complete a demographic and personality survey. We find no time or day differences in behavior, but do find that participants at nights and on weekends are less experienced with online studies; on weekends are less reflective; and at night are less conscientious and more neurotic. These results are largely robust to finer-grained measures of time and day. We also find that those who participated earlier in the course of the study are more experienced, reflective, and agreeable, but less charitable than later participants.

Keywords

Cooperation Honesty Decision-making Time of day MTurk Self-control 

JEL Classification

C80 C90 

Supplementary material

40881_2017_35_MOESM1_ESM.pdf (1.5 mb)
Supplementary material 1 (PDF 1495 kb)

References

  1. Amir, O., Rand, D. G., & Gal, Y. K. (2012). Economic games on the Internet: The effect of $1 stakes. PLoS ONE, 7(2), e31461.CrossRefGoogle Scholar
  2. Arechar, A. A., Molleman, L., & Gachter, S. (2017). Conducting interactive experiments online. Experimental Economics. doi:10.1007/s10683-017-9527-2.Google Scholar
  3. Aviv, A. L., Zelenski, J. M., Rallo, L., & Larsen, R. J. (2002). Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences, 33(3), 487–496.CrossRefGoogle Scholar
  4. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis, 20(3), 351–368. doi:10.1093/pan/mpr057.CrossRefGoogle Scholar
  5. Capraro, V., Jordan, J. J., & Rand, D. G. (2014). Heuristics guide the implementation of social preferences in one-shot Prisoner’s Dilemma experiments. Scientific Reports, 4, 6790.CrossRefGoogle Scholar
  6. Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2016). Intertemporal differences among MTurk worker demographics. https://osf.io/preprints/psyarxiv/8352x.
  7. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.CrossRefGoogle Scholar
  8. Deetlefs, J., Chylinski, M., & Ortmann, A. (2015). MTurk ‘Unscrubbed’: Exploring the good, the ‘super’, and the unreliable on Amazon’s mechanical turk. http://ssrn.com/abstract=2654056.
  9. Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19(4), 25–42.CrossRefGoogle Scholar
  10. Gosling, S. D., Rentfrow, P. J., & Swann, W. B., Jr. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37(6), 504–528.CrossRefGoogle Scholar
  11. Gunia, B. C., Barnes, C. M., & Sah, S. (2014). The morality of larks and owls: Unethical behavior depends on chronotype as well as time of day. Psychological Science, 25(12), 2272–2274.CrossRefGoogle Scholar
  12. Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. doi:10.1007/s10683-011-9273-9.CrossRefGoogle Scholar
  13. Kirby, K. N., Petry, N. M., & Bickel, W. K. (1999). Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. Journal of Experimental Psychology-General, 128(1), 78–87.CrossRefGoogle Scholar
  14. Kouchaki, M., & Smith, I. H. (2014). The morning morality effect: The influence of time of day on unethical behavior. Psychological Science, 25(1), 95–102. doi:10.1177/0956797613498099.CrossRefGoogle Scholar
  15. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk. Judgment and Decision Making, 5(5), 411–419.Google Scholar
  16. Rand, D. G. (2012). The promise of mechanical turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179.CrossRefGoogle Scholar
  17. Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., et al. (2014). Social heuristics shape intuitive cooperation. Nature Communications, 5, 3677.CrossRefGoogle Scholar
  18. Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style influences belief in God. Journal of Experimental Psychology: General, 141(3), 423–428.CrossRefGoogle Scholar
  19. Stagnaro, M. N., Arechar, A. A., & Rand, D. G. (2017). From good institutions to generous citizens: Top-down incentives to cooperate promote subsequent prosociality but not norm enforcement. Cognition. doi:10.1016/j.cognition.2017.01.017.Google Scholar
  20. Peysakhovich, A., Nowak, M. A., & Rand, D. G. (2014). Humans display a ‘cooperative phenotype’ that is domain general and temporally stable. Nature communications, 5, 4939.CrossRefGoogle Scholar

Copyright information

© Economic Science Association 2017

Authors and Affiliations

  1. 1.Department of PsychologyYale UniversityNew HavenUSA
  2. 2.Department of EconomicsYale UniversityNew HavenUSA
  3. 3.School of ManagementYale UniversityNew HavenUSA

Personalised recommendations