Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations
- 3.6k Downloads
Amazon Mechanical Turk is an increasingly popular data source in the organizational psychology research community. This paper presents an evaluation of MTurk and provides a set of practical recommendations for researchers using MTurk.
We present an evaluation of methodological concerns related to the use of MTurk and potential threats to validity inferences. Based on our evaluation, we also provide a set of recommendations to strengthen validity inferences using MTurk samples.
Although MTurk samples can overcome some important validity concerns, there are other limitations researchers must consider in light of their research objectives. Researchers should carefully evaluate the appropriateness and quality of MTurk samples based on the different issues we discuss in our evaluation.
There is not a one-size-fits-all answer to whether MTurk is appropriate for a research study. The answer depends on the research questions and the data collection and analytic procedures adopted. The quality of the data is not defined by the data source per se, but rather the decisions researchers make during the stages of study design, data collection, and data analysis.
The current paper extends the literature by evaluating MTurk in a more comprehensive manner than in prior reviews. Past review papers focused primarily on internal and external validity, with less attention paid to statistical conclusion and construct validity—which are equally important in making accurate inferences about research findings. This paper also provides a set of practical recommendations in addressing validity concerns when using MTurk.
KeywordsAmazon Mechanical Turk MTurk Validity Best practices Recommendations
- Antin, J., & Shaw, A. (2012). Social desirability bias and self-reports of motivation: A study of Amazon Mechanical Turk in the US and India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’12, (pp. 2925–2934).Google Scholar
- Barger, T., Behrend, T. S., Sharek, D. J., & Sinar, E. F. (2011). I-O and the crowd: Frequently asked questions about using Mechanical Turk for research. The Industrial–Organizational Psychologist, 49, 11–18.Google Scholar
- Bergman, M. E., & Jean, V. A. (2016). Where have all the “workers” gone? A critical analysis the unrepresentativeness of our samples relative to the labor market in the industrial–organizational psychology literature. Industrial and Organizational Psychology: Perspectives on Science and Practice, 9, 84–113.CrossRefGoogle Scholar
- Callison-Burch, C., & Dredze, M. (2010). Creating speech and language data with Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT (pp. 1–12).Google Scholar
- Ipeirotis, P. G. (2010). Demographics of Mechanical Turk. NYU Working Paper No.; CEDER-10-01. Retrieved from http://ssrn.com/abstract=1585030
- Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
- Pollack, J., & Aguinis, H. (2013). 2013 JCR journal rankings. Retrieved from https://drive.google.com/file/d/0B68LcC5lXuedZmpXSWFvcTZNck0/edit
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage Learning.Google Scholar
- Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479–491.Google Scholar
- Stone-Romero, E. F. (2011). Research strategies in industrial and organizational psychology: Nonexperimental, quasi-experimental, and randomized experimental research in special purpose and nonspecial purpose settings. In S. Zedeck (Ed.), APA handbook of industrial and organizational psychology (Vol. 1, pp. 37–72). Building and developing the organization Washington, DC: American Psychological Association.Google Scholar
- Welcome to Requester Help. (n.d.). Retrieved from http://requester.mturk.com/help