Abstract
To understand human behavior, social scientists need people and data. In the last decade, Amazon’s Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of human participants and was widely adopted by academics. Yet despite MTurk’s utility, some have questioned whether researchers should continue using the platform on ethical grounds. The brunt of their concern is that people on MTurk are financially insecure, subject to abuse, and earn inhumane wages. We investigated these issues with two representative probability surveys of the U.S. MTurk population (N = 4094). The surveys revealed: (1) the financial situation of people on MTurk mirrors the general population, (2) most participants do not find MTurk stressful or requesters abusive, and (3) MTurk offers flexibility and benefits that most people value above other options for work. People reported it is possible to earn more than $10 per hour and said they would not trade the flexibility of MTurk for less than $25 per hour. Altogether, our data are important for assessing whether MTurk is an ethical place for research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Our data file contains responses from a total of 2277 people. Due to an error when generating the sampling frame, wave 1 of data collection allowed 251 people outside of the U.S. to participate. We excluded responses from these people because we were interested in studying how U.S. MTurk workers feel about the platform.
For the first 562 participants, the sliding answer scale only allowed people to choose a whole number for their answer (e.g., 10%). After repeated messages from participants, however, we changed the scale to accommodate fractional values between whole numbers (e.g., 10.4%). This change was driven by participants who told us that the most accurate answer for the number of HITs they have rejected was a value between 0% and 1%.
References
Anderson, C. A., Allen, J. J., Plante, C., Quigley-McBride, A., Lovett, A., & Rokkum, J. N. (2019). The MTurkification of social and personality psychology. Personality and Social Psychology Bulletin, 45, 842–850. https://doi.org/10.1177/0146167218798821
Arechar, A. A., Kraft-Todd, G. T., & Rand, D. G. (2017). Turking overtime: How participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. Journal of the Economic Science Association, 3, 1–11. https://doi.org/10.1007/s40881017-0035-0
Arechar, A. A., & Rand, D. G. (2021). Turking in the time of COVID. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01588-4
Berg, J. (2015). Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal, 37, 543.
Berg, J. (2016). Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law & Policy Journal, 37(3).
Brenan, M. (2019). Americans Feel Generally Positive About Their Own Finances. Gallup. https://news.gallup.com/poll/249164/americans-feel-generally-positive-own-finances.aspx
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6, 3–5. https://doi.org/10.1177/1745691610393980
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017). Intertemporal differences among MTurk workers: Time-based sample variations and implications for online data collection. SAGE Open, 7, 1–15. https://doi.org/10.1177/2F2158244017712774
Cain Miller, C. (2019). How unpredictable work hours turn families upside down. The New York Times. https://www.nytimes.com/2019/10/16/upshot/unpredictable-job-hours.html
Centers for Disease Control and Prevention (n.d.). Disability impacts all of us. https://www.cdc.gov/ncbddd/disabilityandhealth/infographic-disability-impacts-all.html
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. https://doi.org/10.3758/s13428-013-0365-7
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131–1139. https://doi.org/10.1177/0956797615585115
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51, 2022–2038. https://doi.org/10.3758/s13428-019-01273-7
Coppock, A. (2019). Generalizing from survey experiments conducted on Mechanical Turk: A replication approach. Political Science Research and Methods, 7, 613–628. https://doi.org/10.1017/psrm.2018.10
Damer, E. (2019). Stop using MTurk for research. Prolific Blog. https://blog.prolific.co/stop-using-mturk-for-research/
Difallah, D., Filatova, E., & Ipeirotis, P. (2018). Demographics and dynamics of Mechanical Turk workers. In Proceedings of the eleventh ACM international conference on web search and data mining (pp. 135–143).
Fair Crowd Work. (n.d.). Amazon Mechanical Turk - Review. Retrieved June 9, 2022, from http://faircrowd.work/platform/amazon-mechanical-turk/
FairWork, (2021). FairWork 2021 Annual Report. https://fair.work/wp-content/uploads/sites/131/2022/01/Fairwork-Annual-Report-2021.pdf
Federal Election Commission (2019). Browse disbursements, campaign finance data. https://www.fec.gov/data/disbursements/?data_type=processed&recipient_name=Mechanical+Turk&two_year_transaction_period=2020
Fordsham, N., Moss, A. J., Krumholtz, S., Roggina, T., Jr, Robinson, J., & Litman, L. (2019). Variation among Mechanical Turk Workers Across Time of Day Presents an Opportunity and a Challenge for Research. 10.31234/osf.io/p8bns
Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37, 413–420. https://doi.org/10.1162/COLI_a_00057
Gallup. (n.d.). Work and Workplace. Gallup.Com. Retrieved August 10, 2022, from https://news.gallup.com/poll/1720/Work-Work-Place.aspx
Graham, F. (2010). Crowdsourcing work: Labour on demand or digital sweatshop? BBC News. https://www.bbc.com/news/business-11600902
Gray, M. L., & Suri, S. (2019). Ghost work: How to stop Silicon Valley from building a new global underclass. Houghton Mifflin Harcourt.
Grim, R., & Lacey, A. (2020). Pete Buttigieg’s campaign used notoriously low-paying gig-work platform for polling. The Intercept. https://theintercept.com/2020/01/16/pete-buttigieg-amazon-mechanical-turk-gig-workers/
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018). A data-driven analysis of workers' earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–14). https://doi.org/10.1145/3173574.3174023
Hauser, D. J., Paolacci, G., & Chandler, J. J. (2019). Common concerns with MTurk as a participant pool: Evidence and solutions. In F. R. Kardes, P. M. Herr, & N. Schwarz (Eds.), Handbook of Research Methods in Consumer Psychology. Routledge.
Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400–407. https://doi.org/10.3758/s13428-015-0578-z
Hitlin, P. (2016). Research in the crowdsourcing age, a case study. Pew Research Center. Available at: https://www.pewresearch.org/internet/2016/07/11/research-in-the-crowdsourcing-age-a-case-study/
Horton, J. J. (2011). The condition of the Turking class: Are online employers fair and honest? Economics Letters, 111, 10–12. https://doi.org/10.1016/j.econlet.2010.12.007
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425. https://doi.org/10.1007/s10683-011-9273-9
Ingraham, C. (2019). Nine days on the road. Average commute time reached a new record last year. The Washington Post. https://www.washingtonpost.com/business/2019/10/07/nine-days-road-average-commute-timereached-new-record-last-year/
Ipeiriotis, P. (2010). The demographics of Mechanical Turk (NYU Working Paper No. CEDER-10-01). Retrieved from SSRN: https://ssrn.com/abstract=1585030
Kaplan, T., Saito, S., Hara, K., & Bigham, J. P. (2018). Striving to earn more: A survey of work strategies and tool use among crowd workers. In Sixth AAAI Conference on Human Computation and Crowdsourcing.
Kennedy, C., & Hartig, H. (2019). Response rates in telephone surveys have resumed their decline. Pew Research Center. Available at: https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/
Litman, L., & Robinson, J. (2020a). Introduction. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon Mechanical Turk and beyond (pp. 1–26). Sage Academic Publishing.
Litman, L., & Robinson, J. (2020b). Conducting ethical online research: A data-driven approach. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon Mechanical Turk and beyond (pp. 234–263). Sage Academic Publishing. Thousand Oaks.
Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US and India based workers on Mechanical Turk. Behavior Research Methods, 47, 519–528. https://doi.org/10.3758/s13428-014-0483-x
Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49, 433–442. https://doi.org/10.3758/s13428-016-0727-z
Litman, L., Robinson, J., Rosen, Z., Rosenzweig, C., Waxman, J., & Bates, L. M. (2020a). The persistence of pay inequality: The gender wage gap in an anonymous online labor market. PLOS ONE, 15(2), e0229383. https://doi.org/10.1371/journal.pone.0229383
Litman, L., Robinson, J., & Rosenzweig, C. (2020b). Sampling Mechanical Turk workers: Problems and solutions. In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon Mechanical Turk and beyond (pp. 121–146). Sage Academic Publishing. Thousand Oaks.
Marken, S. (2018). Still listening: The state of telephone surveys. Gallup [blog post]. Available at: https://news.gallup.com/opinion/methodology/225143/listening-state-telephone-surveys.aspx
Meyers, E. A., Walker, A. C., Fugelsang, J. A., & Koehler, D. J. (2020). Reducing the number of non-naïve participants in Mechanical Turk samples. Methods in Psychology, 3, 100032. https://doi.org/10.1016/j.metip.2020.100032
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2, 109–138. https://doi.org/10.1017/XPS.2015.19
Newman, A. (2019). I found work on an Amazon website. I made 97 cents an hour. The New York Times. https://www.nytimes.com/interactive/2019/11/15/nyregion/amazon-mechanical-turk.html
Norman, J. (2018). Four in 10 U.S. Workers Think They Are Underpaid. Gallup. https://news.gallup.com/poll/241682/four-workers-think-underpaid.aspx
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. https://doi.org/10.1016/j.jesp.2017.01.006
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. https://doi.org/10.3758/s13428-013-0434-y
Rao, S., & Michel, A. (2010). ProPublica’s Guide to Mechanical Turk. ProPublica. https://www.propublica.org/article/propublicas-guide-to-mechanical-turk
Robinson, J., Litman, L., & Rosenzweig, C. (2020a). Who are the Mechanical Turk workers? In L. Litman & J. Robinson (Eds.), Conducting online research on Amazon Mechanical Turk and beyond (pp. 121–146). Sage Academic Publishing.
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLoS ONE, 14(12), e0226394. https://doi.org/10.1371/journal.pone.0226394
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in Mechanical Turk. In CHI'10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). https://doi.org/10.1145/1753846.1753873
Sassenberg, K., & Ditrich, L. (2019). Research in social psychology changed between 2011 and 2016: Larger sample sizes, more self-report measures, and more online studies. Advances in Methods and Practices in Psychological Science, 2, 107–114. https://doi.org/10.1177/2515245919838781
Schneider, D., & Harknett, K. (2019). It’s about time: How work schedule instability matters for workers, families, and racial inequality. The Shift Project. https://shift.berkeley.edu/its-about-time-how-work-schedule-instability-matters-for-workers-families-and-racial-inequality/
Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515–530. https://doi.org/10.1037/0022-3514.51.3.515
Semuels, A. (2018). The online hell of Amazon Mechanical Turk. The Atlantic. https://www.theatlantic.com/business/archive/2018/01/amazon-mechanical-turk/551192/
TurkerView. (2019). Writer who never learned to drive works for Uber. Makes $0.97/hr. https://blog.turkerview.com/writer-who-never-learned-to-drive-works-for-uber/
U.S. Census Bureau. (2018). Current population survey, selected characteristics of households by total money income. [Data set]. https://www.census.gov/data/tables/time-series/demo/income-poverty/cps-hinc/hinc-01.html
Williamson, V. (2016). On the ethics of crowdsourced research. PS. Political Science & Politics, 49(1), 77–81. https://doi.org/10.1017/S104909651500116X
Yeager, D. S., Krosnick, J. A., Visser, P. S., Holbrook, A. L., & Tahk, A. M. (2019). Moderation of classic social psychological effects by demographics in the U.S. adult population: New opportunities for theoretical advancement. Journal of Personality and Social Psychology, 117, e84–e99. https://doi.org/10.1037/pspa0000171
Open Practices Statement
Study 1 was preregistered (see here: https://osf.io/cwde4) but Study 2 was not. All data, materials, and analysis code are available on the Open Science Framework: https://osf.io/apved/?view_only=37f51a62db6144a2950646975887bac9
Funding
The funder, Prime Research Solutions, provided support in the form of salaries for authors [A.M, C.R., J.R., S.J. and L.L.]. All decisions about study design, data collection and analysis, decision to publish, and preparation of the manuscript were made by the research team. The specific role of each author is articulated in the ‘author contributions’ section.
Author information
Authors and Affiliations
Contributions
Corresponding author
Ethics declarations
Competing interests
The authors of this manuscript have the following potential competing interests: all authors are employed at CloudResearch (formerly TurkPrime). CloudResearch provides online research tools and services, including tools that allow researchers to run studies on Mechanical Turk.
CloudResearch’s MTurk ToolKit was used to source Mechanical Turk participants, and the CloudResearch database was used to query some of the data.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(DOCX 75 kb)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Moss, A.J., Rosenzweig, C., Robinson, J. et al. Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. Behav Res 55, 4048–4067 (2023). https://doi.org/10.3758/s13428-022-02005-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3758/s13428-022-02005-0