Abstract
Crowdsourcing is an increasingly popular method for organizational and occupational health research. Amazon’s Mechanical Turk (MTurk) is at the forefront of this trend, but few studies have examined the labor market characteristics of MTurk workers (Turkers) or results of organizational and occupational health data as compared to published benchmarks. To address these gaps, we review the current MTurk literature and present the results of a multi-wave study of Turker labor characteristics and organizational and occupational health variable relationships. We found Turkers to be broadly distributed across the labor market, indicating MTurk is a viable option for both generalizable and understudied samples, as well as a source for targeted occupational sectors. Additionally, we found effect size magnitudes to be comparable to published benchmarks, and data displayed high levels of test-retest reliability and stability of relationships across time. Our results support the use of MTurk as a viable source for organizational and occupational health research assuming general methodological concerns and validity threats are addressed (see Cheung et al. Journal of Business and Psychology 1–15, 2016).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Allen, T. D., Johnson, R. C., Saboe, K. N., Cho, E., Dumani, S., & Evans, S. (2012). Dispositional variables and work–family conflict: A meta-analysis. Journal of Vocational Behavior, 80(1), 17–26. https://doi.org/10.1016/j.jvb.2011.04.004.
Amstad, F. T., Meier, L. L., Fasel, U., Elfering, A., & Semmer, N. K. (2011). A meta-analysis of work–family conflict and various outcomes with a special emphasis on cross-domain versus matching-domain relations. Journal of Occupational Health Psychology, 16(2), 151–169. https://doi.org/10.1037/a0022170.
Armon, G., Shirom, A., Berliner, S., Shapira, I., & Melamed, S. (2008). A prospective study of the association between obesity and burnout among apparently healthy men and women. Journal of Occupational Health Psychology, 13(1), 43–57. https://doi.org/10.1037/1076-8998.13.1.43.
Bakker, M., Hartgerink, C. H., Wicherts, J. M., & van der Maas, H. L. (2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27(8), 1069–1077. https://doi.org/10.1177/0956797616647519.
Banks, M. H., Clegg, C. W., Jackson, P. R., Kemp, N. J., Stafford, E. M., & Wall, T. D. (1980). The use of the general health questionnaire as an indicator of mental health in occupational studies. Journal of Occupational Psychology, 53, 187–194. https://doi.org/10.1111/j.2044-8325.1980.tb00024.x.
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 1–14. https://doi.org/10.3758/s13428-011-0081-0.
Bergman, M. E., & Jean, V. A. (2016). Where have all the “workers” gone? A critical analysis of the unrepresentativeness of our samples relative to the labor market in the industrial–organizational psychology literature. Industrial and Organizational Psychology, 9(1), 84–113. https://doi.org/10.1017/iop.2015.70.
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's mechanical Turk. Political Analysis, 20(3), 351–368. https://doi.org/10.1093/pan/mpr057.
Buhrmester, M., Kwang, T., & Gosling, S. (2011). Amazon's mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. https://doi.org/10.1177/1745691610393980.
Bureau of Labor Statistics. (2005). Contingent and alternative employment arrangements. February 2005 (USDL 05–1433). Washington, DC: U.S. Department of Labor.
Carlson, D. S., Kacmar, K. M., & Williams, L. J. (2000). Construction and initial validation of a multidimensional measure of work–family conflict. Journal of Vocational Behavior, 56(2), 249–276. https://doi.org/10.1006/jvbe.1999.1713.
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2016). Amazon mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business and Psychology, 1–15. https://doi.org/10.1007/s10869-016-9458-5.
Christenson, D. P., & Glick, D. M. (2013). Crowdsourcing panel studies and real-time experiments in MTurk. The Political Methodologist, 20(2), 27–32.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Erlbaum.
Daly, T. M., & Nataraajan, R. (2015). Swapping bricks for clicks: Crowdsourcing longitudinal data on Amazon Turk. Journal of Business Research, 68(12), 2603–2609. https://doi.org/10.1016/j.jbusres.2015.05.001.
Gardner, R. M., Brown, D. L., & Boice, R. (2012). Using Amazon's mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. Body Image, 9(4), 532–534. https://doi.org/10.1016/j.bodyim.2012.06.006.
Goldberg, D. (1978). Manual of the general health questionnaire. Windsor: NFER-Nelson.
Goldberg, D. P., & Hillier, V. F. (1979). A scaled version of the general health questionnaire. Psychological Medicine, 9(1), 139–145. https://doi.org/10.1017/S0033291700021644.
Goldberg, D., & Williams, P. (2006). A user's guide to the general health questionnaire. GL assessment.
Grawitch, M. J., Maloney, P. W., Barber, L. K., & Mooshegian, S. E. (2013). Examining the nomological network of satisfaction with work–life balance. Journal of Occupational Health Psychology, 18(3), 276–284. https://doi.org/10.1037/a0032754.
Greenhaus, J. H., & Beutell, N. J. (1985). Sources of conflict between work and family roles. Academy of Management Review, 10(1), 76–88. https://doi.org/10.2307/258214.
Halbesleben, J. R. (2010). A meta-analysis of work engagement: Relationships with burnout, demands, resources, and consequences. Work engagement: A handbook of essential theory and research, 8, 102–117.
Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. https://doi.org/10.3758/s13428-015-0578-z.
Howard, J., Gagne, M., Morin, A. J., & Van den Broeck, A. (2016). Motivation profiles at work: A self-determination theory approach. Journal of Vocational Behavior, 95, 74–89. https://doi.org/10.1016/j.jvb.2016.07.004.
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. https://doi.org/10.1007/s10869-011-9231-8.
Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845. https://doi.org/10.1037/a0038510.
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8(2), 142–164. https://doi.org/10.1017/iop.2015.13.
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's mechanical Turk. Behavior Research Methods, 44, 1–23. https://doi.org/10.3758/s13428-011-0124-6.
Matthews, R. A., Mills, M. J., Trout, R. C., & English, L. (2014a). Family-supportive supervisor behaviors, work engagement, and subjective well-being: A contextually dependent mediated process. Journal of Occupational Health Psychology, 19(2), 168–181. https://doi.org/10.1037/a0036012.
Matthews, R. A., Wayne, J. H., & Ford, M. T. (2014b). A work–family conflict/subjective well-being process model: A test of competing theories of longitudinal effects. Journal of Applied Psychology, 99(6), 1173–1187. https://doi.org/10.1037/a0036674.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085.
Mechanical Turk Monitor (2017). http://www.mturk-tracker.com.
Melamed, S., Shirom, A., Toker, S., Berliner, S., & Shapira, I. (2006). Burnout and risk of cardiovascular disease: Evidence, possible causal paths, and promising research directions. Psychological Bulletin, 132(3), 327–353. https://doi.org/10.1037/0033-2909.132.3.327.
Michel, J. S., Kotrba, L. M., Mitchelson, J. K., Clark, M. A., & Baltes, B. B. (2011). Antecedents of work–family conflict: A meta‐analytic review. Journal of Organizational Behavior, 32(5), 689–725. https://doi.org/10.1002/job.695.
Michel, J. S., Hartman, P., O'Neill, S. K., Lorys, A., & Chen, P. Y. (2016). Sample adequacy and implications for occupational health psychology research. Industrial and Organizational Psychology, 9(1), 193–197. https://doi.org/10.1017/iop.2015.135.
Nahrgang, J. D., Morgeson, F. P., & Hofmann, D. A. (2011). Safety at work: A meta-analytic investigation of the link between job demands, job resources, burnout, engagement, and safety outcomes. Journal of Applied Psychology, 96(1), 71–94. https://doi.org/10.1037/a0021484.
Netemeyer, R. G., Boles, J. S., & McMurrian, R. (1996). Development and validation of work–family conflict and family–work conflict scales. Journal of Applied Psychology, 81(4), 400–410.
Nicholson, J. R. (2015). Temporary help Workers in the U.S. labor market. (ESA issue brief 03-15). Washington, DC: U.S. Department of Commerce.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716.
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding mechanical Turk as a participant pool. Curent Directions in Psychological Science, 23(3), 184–188. https://doi.org/10.1177/0963721414531598.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk. Judgment and Decision making, 5(5), 411–419.
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. https://doi.org/10.3758/s13428-013-0434-y.
Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903. https://doi.org/10.1037/0021-9010.88.5.879.
Reichl, C., Leiter, M. P., & Spinath, F. M. (2014). Work–nonwork conflict and burnout: A meta-analysis. Human Relations, 67(8), 979–1005. https://doi.org/10.1177/0018726713509857.
Rich, B. L., Lepine, J. A., & Crawford, E. R. (2010). Job engagement: Antecedents and effects on job performance. Academy of Management Journal, 53(3), 617–635. https://doi.org/10.5465/AMJ.2010.51468988.
Schaufeli, W. B., Bakker, A. B., & Salanova, M. (2006). The measurement of work engagement with a short questionnaire a cross-national study. Educational and Psychological Measurement, 66(4), 701–716. https://doi.org/10.1177/0013164405282471.
Schaufeli, W. B., Taris, T. W., & Van Rhenen, W. (2008). Workaholism, burnout, and work engagement: Three of a kind or three different kinds of employee well-being? Applied Psychology, 57(2), 173–203. https://doi.org/10.1111/j.1464-0597.2007.00285.x.
Schaufeli, W. B., Bakker, A. B., & Van Rhenen, W. (2009). How changes in job demands and resources predict burnout, work engagement, and sickness absenteeism. Journal of Organizational Behavior, 30(7), 893–917. https://doi.org/10.1002/job.595.
Schleider, J. L., & Weisz, J. R. (2015). Using mechanical Turk to study family processes and youth mental health: A test of feasibility. Journal of Child and Family Studies, 24(11), 3235–3246. https://doi.org/10.1007/s10826-015-0126-6.
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using mechanical Turk to study clinical populations. Clinical Psychological Science, 1(2), 213–220. https://doi.org/10.1177/2167702612469015.
Shirom, A., & Melamed, S. (2006). A comparison of the construct validity of two burnout measures in two groups of professionals. International Journal of Stress Management, 13, 176–200. https://doi.org/10.1037/1072-5245.13.2.176.
Shockley, K. M., & Singla, N. (2011). Reconsidering work—family interactions and satisfaction: A meta-analysis. Journal of Management, 37(3), 861–886. https://doi.org/10.1177/0149206310394864.
Sprouse, J. (2011). A validation of Amazon mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155–167. https://doi.org/10.3758/s13428-010-0039-7.
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251. https://doi.org/10.1037/0033-2909.87.2.245.
Toker, S., Shirom, A., Shapira, I., Berliner, S., & Melamed, S. (2005). The association between burnout, depression, anxiety, and inflammation biomarkers: C-reactive protein and fibrinogen in men and women. Journal of Occupational Health Psychology, 10(4), 344–362. https://doi.org/10.1037/1076-8998.10.4.344.
Torpey E. (2015). Clusters, pathways, and BLS: Connecting career information. Washington, DC: Bureau of Labor Statistics, U.S. Department of Labor. Retrieved from http://www.bls.gov/careeroutlook/2015/article/career-clusters.htm.
Wayne, J. H., Butts, M. M., Casper, W. J., & Allen, T. D. (2017). In search of balance: A conceptual and empirical integration of multiple meanings of work–family balance. Personnel Psychology, 70(1), 167–210. https://doi.org/10.1111/peps.12132.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Michel, J.S., O’Neill, S.K., Hartman, P. et al. Amazon’s Mechanical Turk as a Viable Source for Organizational and Occupational Health Research. Occup Health Sci 2, 83–98 (2018). https://doi.org/10.1007/s41542-017-0009-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41542-017-0009-x