A Study of Daily Sample Composition on Amazon Mechanical Turk
- 2.6k Downloads
Amazon Mechanical Turk (AMT) has become a powerful tool for social scientists due to its inexpensiveness, ease of use, and ability to attract large numbers of workers. While the subject pool is diverse, there are numerous questions regarding the composition of the workers as a function of when the “Human Intelligence Task”(HIT) is posted. Given the “queue” nature of HITs and the disparity in geography of participants, it is natural to wonder whether HIT posting time/day can have an impact on the population that is sampled. We address this question using a panel survey on AMT and show (surprisingly) that except for gender, there is no statistically significant difference in terms of demographics characteristics as a function of HIT posting time.
KeywordsSurvey Response Panel Survey Subject Pool Inverse Gaussian Distribution Validation Code
Unable to display preview. Download preview PDF.
- 2.Chilton, L.B., Horton, J.J., Miller, R.C., Azenkot, S.: Task search in a human computation market. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP 2010, pp. 1–9. ACM, New York (2010). http://doi.acm.org/10.1145/1837885.1837889
- 4.Henrich, J., Heine, S.J., Norenzayan, A.: The weirdest people in the world? The Behavioral and Brain Sciences 33(2–3), 61–83; discussion 83–135 (2010)Google Scholar
- 5.Hogg, T., Lerman, K., Smith, L.M.: Stochastic models predict user behavior in social media. HUMAN 2(1), 25–39 (2013). http://ojs.scienceengineering.org/index.php/human/article/view/72 Google Scholar
- 8.Komarov, S., Reinecke, K., Gajos, K.Z.: Crowdsourcing performance evaluations of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2013, pp. 207–216. ACM, New York (2013). http://doi.acm.org/10.1145/2470654.2470684
- 9.Lease, M., Hullman, J., Bigham, J.P., Bernstein, M.S., Kim, J., Lasecki, W., Bakhshi, S., Mitra, T., Miller, R.C.: Mechanical turk is not anonymous. SSRN Scholarly Paper ID 2228728, Social Science Research Network, Rochester, NY, March 2013. http://papers.ssrn.com/abstract=2228728
- 10.Mason, W., Suri, S.: Conducting behavioral research on amazon’s mechanical turk. Behavior Research Methods 44(1), 1–23 (2012). http://link.springer.com/article/10.3758/s13428-011-0124-6 CrossRefGoogle Scholar
- 11.Oppenheimer, D.M., Meyvis, T., Davidenko, N.: Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology 45(4), 867–872 (2009). http://www.sciencedirect.com/science/article/pii/S0022103109000766 CrossRefGoogle Scholar
- 13.Rand, D.G.: The promise of mechanical turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology 299, 172–179 (2012). http://www.sciencedirect.com/science/article/pii/S0022519311001330 CrossRefMathSciNetGoogle Scholar