Behavior Research Methods

, Volume 47, Issue 2, pp 519–528 | Cite as

The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk

  • Leib Litman
  • Jonathan Robinson
  • Cheskie Rosenzweig
Article

Abstract

In this study, we examined data quality among Amazon Mechanical Turk (MTurk) workers based in India, and the effect of monetary compensation on their data quality. Recent studies have shown that work quality is independent of compensation rates, and that compensation primarily affects the quantity but not the quality of work. However, the results of these studies were generally based on compensation rates below the minimum wage, and far below a level that was likely to play a practical role in the lives of workers. In this study, compensation rates were set around the minimum wage in India. To examine data quality, we developed the squared discrepancy procedure, which is a task-based quality assurance approach for survey tasks whose goal is to identify inattentive participants. We showed that data quality is directly affected by compensation rates for India-based participants. We also found that data were of a lesser quality among India-based than among US participants, even when optimal payment strategies were utilized. We additionally showed that the motivation of MTurk users has shifted, and that monetary compensation is now reported to be the primary reason for working on MTurk, among both US- and India-based workers. Overall, MTurk is a constantly evolving marketplace where multiple factors can contribute to data quality. High-quality survey data can be acquired on MTurk among India-based participants when an appropriate pay rate is provided and task-specific quality assurance procedures are utilized.

Keywords

Amazon Mechanical Turk Research methods Data quality 

References

  1. Antin, J., & Shaw, A. (2012). Social desirability bias and self-reports of motivation: A study of Amazon Mechanical Turk in the US and India. In CHI ’12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2925–2934). New York, NY: ACM.Google Scholar
  2. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351–368.CrossRefGoogle Scholar
  3. Bohannon, J. (2011). Human subject research: Social science for pennies. Science, 334, 307. doi:10.1126/science.334.6054.307 CrossRefPubMedGoogle Scholar
  4. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980 CrossRefGoogle Scholar
  5. Bureau of Democracy, Human Rights and Labor. (2012). Country Reports on Human Rights Practices, India. Retrieved December 22, 2013, from U.S. Department of State website, www.state.gov/j/drl/rls/hrrpt/2012humanrightsreport/index
  6. Chandler, J., Mueller, P., & Paolacci, G. (2013). Methodological concerns and advanced uses of crowdsourcing in psychological research. Manuscript submitted for publication. http://www.jessechandler.com/uploads/2/8/0/5/2805897/mturk_adv_methods.pdf
  7. Chouliarakis, G., & Correa-López, M. (2014). A fair wage model of unemployment with inertia in fairness perceptions. Oxford Economic Papers, 66(1), 88–114.CrossRefGoogle Scholar
  8. DeScioli, P., Christner, J., & Kurzban, R. (2011). The omission strategy. Psychological Science, 22, 442–446.CrossRefPubMedGoogle Scholar
  9. Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are your participants gaming the system? Screening Mechanical Turk Workers. In CHI ’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2399–2402). New York, NY: ACM.Google Scholar
  10. Eriksson, K., & Simpson, B. (2010). Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making, 5, 159–163.Google Scholar
  11. Falk, A., Fehr, E., & Zehnder, C. (2006). Fairness perceptions and reservation wages—The behavioral effects of minimum wage laws. The Quarterly Journal of Economics, 121, 1347–1381.Google Scholar
  12. Faridani, S., Hartmann, B., & Ipeirotis, P. G. (2011). What’s the right price? Pricing tasks for finishing on time. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence (pp. 26–31). Menlo Park, CA: AAAI.Google Scholar
  13. Gardner, R. M., Brown, D. L., & Boice, R. (2012). Using Amazon’s Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. Body Image, 9, 532–534.CrossRefPubMedGoogle Scholar
  14. Government of India, Ministry of Labor and Employment. (2011).Second annual report to the people on employment. Retrieved December 22, 2013, from the Government of India website, dget.gov.in/publications/annualreportemployment2011.pdf.Google Scholar
  15. Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In EC ’10: Proceedings of the 11th ACM Conference on Electronic Commerce (pp. 209–218). New York, NY: ACM. doi:10.1145/1807342.1807376 Google Scholar
  16. Ipeirotis, P. (2010). Demographics of Mechanical Turk [Working Article]. New York University. http://hdl.handle.net/2451/29585, http://scholar.google.com/scholar?cluster=8873729056826117132&hl=en&as_sdt=0,33
  17. Jenkins, G. D. J., Mitra, A., Gupta, N., & Shaw, J. D. (1998). Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology, 83, 777–787.CrossRefGoogle Scholar
  18. John, O. P., Robins, R. W., & Pervin, L. A. (2008). Handbook of personality, third edition: Theory and research. New York, NY: Guilford Press.Google Scholar
  19. Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips: Using Amazon’s Mechanical Turk to increase student-faculty collaborative research. Teaching of Psychology, 39, 245–251.CrossRefGoogle Scholar
  20. Jones, D. N., & Paulhus, D. L. (2011). The role of impulsivity in the Dark Triad of personality. Personality and Individual Differences, 51, 679–682.CrossRefGoogle Scholar
  21. Kazai, G., Kamps, J., & Milic-Frayling, N. (2013). An analysis of human factors and label accuracy in crowdsourcing relevance judgements. Information Retrieval Journal, 16, 138–178. doi:10.1007/s10791-012-9205-0 CrossRefGoogle Scholar
  22. Marge, M., Banerjee, S., & Rudnicky, A. I. (2010). Using the Amazon Mechanical Turk for transcription of spoken language. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (pp. 5270–5273). Los Alamitos, CA: IEEE Press. doi:10.1109/ICASSP.2010.5494979 Google Scholar
  23. Mason, W., & Watts, D. J. (2010). Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2), 100–108.Google Scholar
  24. Nyhan, B., & Reifler, J. (2011). Opening the political mind? The effects of self-affirmation and graphical information on factual misperceptions. Unpublished manuscript. Hanover, NH: Dartmouth College.Google Scholar
  25. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.CrossRefGoogle Scholar
  26. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
  27. Paulhus, D. L., & Carey, J. M. (2011). The FAD-Plus: measuring lay beliefs regarding free will and related constructs. Journal of Personality Assessment, 93, 96–104.CrossRefPubMedGoogle Scholar
  28. Rani, U., & Belser, P. (2012). The effectiveness of minimum wages in developing countries: The case of India. International Journal of Labour Research, 4, 45–66.Google Scholar
  29. Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in Mechanical Turk. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). New York, NY: ACM.Google Scholar
  30. Schmitt, D. P., Allik, J., McCrae, R. R., & Benet-Martinez, V. (2007). The geographic distribution of Big Five Personality traits: Patterns and profiles of human self-description across 56 nations. Journal of Cross-Cultural Psychology, 38, 173–212.CrossRefGoogle Scholar
  31. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.CrossRefGoogle Scholar
  32. Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style influences belief in God. Journal of Experimental Psychology: General, 141, 423–428. doi:10.1037/a0025391 CrossRefGoogle Scholar
  33. Silberman, M. S., Ross, J., Irani, L., & Tomlinson, B. (2010). Sellers’ problems in human computation markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation (pp. 18–21). New York, NY: ACM.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  • Leib Litman
    • 1
  • Jonathan Robinson
    • 2
  • Cheskie Rosenzweig
    • 1
  1. 1.Department of PsychologyLander CollegeFlushingUSA
  2. 2.Department of Computer ScienceLander CollegeFlushingUSA

Personalised recommendations