Evaluating Amazon’s Mechanical Turk for psychological research on the symbolic control of attention

  • Joseph R. Pauszek
  • Pedro Sztybel
  • Bradley S. Gibson


The use of online crowdsourcing services like Amazon’s Mechanical Turk (AMT) as a method of collecting behavioral data online has become increasingly popular in recent years. A growing body of contemporary research has empirically validated the use of AMT as a tool in psychological research by replicating a wide range of well-established effects that have been previously reported in controlled laboratory studies. However, the potential for AMT to be used to conduct spatial cuing experiments has yet to be investigated in depth. Spatial cuing tasks are typically very basic in terms of their stimulus complexity and experimental testing procedures, thus making them ideal for remote testing online that requires minimal task instruction. Studies employing the spatial cuing paradigm are typically aimed at unveiling novel facets of the symbolic control of attention, which occurs whenever observers orient their attention through space in accordance with the meaning of a spatial cue. Ultimately, the present study empirically validated the use of AMT to study the symbolic control of attention by successfully replicating four hallmark effects reported throughout the visual attention literature: the left/right advantage, cue type effect, cued axis effect, and cued endpoint effect. Various recommendations for future endeavors using AMT as a means of remotely collecting behavioral data online are also provided. In sum, the present study provides a crucial first step toward establishing a novel tool for conducting psychological research that can be used to expedite not only our own scientific contributions, but also those of our colleagues.


Amazon Mechanical Turk Crowdsourcing Visual attention Spatial cuing 

Supplementary material

13428_2016_847_MOESM1_ESM.pdf (283 kb)
(PDF 283 kb)


  1. Abrams, J., Nizam, A., & Carrasco, M. (2012). Isoeccentric locations are not equivalent: The extent of the vertical meridian asymmetry. Vision Research, 52, 70–78. doi:10.1016/j.visres.2011.10.016 CrossRefPubMedGoogle Scholar
  2. Bates, J. A., & Lanza, B. A. (2013). Conducting psychology student research via the Mechanical Turk crowdsourcing service. North American Journal of Psychology, 15, 385–394.Google Scholar
  3. Becker, M., Nevins, A., & Levine, J. (2012). Asymmetries in generalizing alternations to and from initial syllables. Language, 88, 231–268.CrossRefGoogle Scholar
  4. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980 CrossRefPubMedGoogle Scholar
  5. Carrasco, M. (2014). Spatial covert attention: Perceptual modulation. In A. C. Nobre & S. Kastner (Eds.), The Oxford handbook of attention (pp. 183–230). New York, NY: Oxford University Press.Google Scholar
  6. Connolly, M., & Van Essen, D. C. (1984). The representation of the visual field in parvicellular and magnocellular layers of the lateral geniculate nucleus in the macaque monkey. Journal of Comparative Neurology, 226, 544–564.CrossRefPubMedGoogle Scholar
  7. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8(e57410), 1–18. doi:10.1371/journal.pone.0057410 Google Scholar
  8. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analyses. New York, NY: Routledge.Google Scholar
  9. Curcio, C. A., & Allen, K. A. (1990). Topography of ganglion cells in human retina. The Journal of Comparative Neurology, 300, 5–25.CrossRefPubMedGoogle Scholar
  10. Curcio, C. A., Sloan, K. R., Packer, O., Hendrickson, A. E., & Kalina, R. E. (1987). Distribution of cones in human and monkey retina: Individual variability and radial asymmetry. Science, 236, 579–582.CrossRefPubMedGoogle Scholar
  11. Curcio, C. A., Sloan, K. R., Kalina, R. E., & Hendrickson, A. E. (1990). Human photoreceptor topography. The Journal of Comparative Neurology, 292, 497–523.CrossRefPubMedGoogle Scholar
  12. Enochson, K., & Culbertson, J. (2015). Collecting psycholinguistic response time data using Amazon Mechanical Turk. PLoS ONE, 10(e116946), 1–17. doi:10.1371/journal.pone.0116946 Google Scholar
  13. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. doi:10.3758/BF03193146 CrossRefPubMedGoogle Scholar
  14. Fine, A. B., Jaeger, T. F., Farmer, T. A., & Qian, T. (2013). Rapid expectation adaptation during syntactic comprehension. PLoS ONE, 8(e77661), 1–18. doi:10.1371/journal.pone.0077661 Google Scholar
  15. Forster, K. I., & Forster, J. C. (2003). DMDX: A Windows display program with millisecond accuracy. Behavior Research Methods, Instruments, & Computers, 35, 116–124. doi:10.3758/BF03195503 CrossRefGoogle Scholar
  16. Gibson, B. S., & Davis, G. J. (2011). Grounding spatial language in the motor system: Reciprocal interactions between conceptual control and spatial orienting. Visual Cognition, 19, 79–116.CrossRefGoogle Scholar
  17. Gibson, B. S., & Kingstone, A. F. (2005). Visual attention and the semantics of space: Beyond central and peripheral cues. Paper presented at the 46th Annual Meeting of the Psychonomic Society, Toronto, CA.Google Scholar
  18. Gibson, B. S., & Kingstone, A. (2006). Visual attention and the semantics of space: Beyond central and peripheral cues. Psychological Science, 17, 622–627.CrossRefPubMedGoogle Scholar
  19. Gibson, B. S., & Sztybel, P. (2014). The spatial semantics of symbolic attention control. Current Directions in Psychological Science, 23, 271–276.CrossRefGoogle Scholar
  20. Gibson, B. S., Scheutz, M., & Davis, G. J. (2009). Symbolic control of visual attention: Semantic constraints of the spatial distribution of attention. Attention, Perception, & Psychophysics, 71, 363–374. doi:10.3758/APP.71.2.363 CrossRefGoogle Scholar
  21. Gibson, E., Piantadosi, S., & Fedorenko, K. (2011). Using Mechanical Turk to obtain and analyze English acceptability judgments. Language and Linguistics Compass, 5, 509–524.CrossRefGoogle Scholar
  22. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61–83. doi:10.1017/S0140525X0999152X CrossRefPubMedGoogle Scholar
  23. Ipeirotis, P. G. (2010). Demographics of Mechanical Turk. Center for Digital Economy Research Working Papers, 10. Available at http://hdl.handle.net/2451/29585
  24. Kemmerer, D. (2006). The semantics of space: Integrating linguistic typology and cognitive neuroscience. Neuropsychologia, 44, 1607–1621.CrossRefPubMedGoogle Scholar
  25. Kobayashi, H., & Kohshima, S. (1997). Unique morphology of the human eye. Nature, 387, 767–768.CrossRefPubMedGoogle Scholar
  26. Kobayashi, H., & Kohshima, S. (2001a). Evolution of the human eye as a device for communication. In T. Matsuzawa (Ed.), Primates origins of human cognition and behavior (pp. 383–401). Tokyo, Japan: Springer.Google Scholar
  27. Kobayashi, H., & Kohshima, S. (2001b). Unique morphology of the human eye and its adaptive meaning: Comparative studies on external morphology of the primate eye. Journal of Human Evolution, 40, 419–435.CrossRefPubMedGoogle Scholar
  28. Logan, G. D. (1995). Linguistic and conceptual control of visual spatial attention. Cognitive Psychology, 28, 103–174. doi:10.1006/cogp.1995.1004 CrossRefPubMedGoogle Scholar
  29. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23. doi:10.3758/s13428-011-0124-6 CrossRefPubMedGoogle Scholar
  30. Maunsell, J. H. R., & Van Essen, D. C. (1987). Topographic organization of the middle temporal visual area in the macaque monkey: Representational biases and the relationship to callosal connections and myeloarchitectonic boundaries. Journal of Comparative Neurology, 266, 535–555.CrossRefPubMedGoogle Scholar
  31. Myers, D. G. (1983). Social psychology. New York, NY: McGraw-Hill Education.Google Scholar
  32. Nomura, M., Katahata, M., & Hashiya, K. (2005). Visual orienting occurs asymmetrically in horizontal vs. vertical plans. Psychologia, 48, 205–217.CrossRefGoogle Scholar
  33. Ossandón, J. P., Onat, S., & König, P. (2014). Spatial biases in viewing behavior. Journal of Vision, 14(2), 20:1–26. doi:10.1167/14.2.20
  34. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188. doi:10.1177/0963721414531598 CrossRefGoogle Scholar
  35. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
  36. Pauszek, J. R., & Gibson, B. S. (2016). High spatial validity is not sufficient to elicit voluntary shifts of attention. Attention, Perception, & Psychophysics, 78, 2110–2123. doi:10.3758/s13414-016-1097-4 CrossRefGoogle Scholar
  37. Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3–25. doi:10.1080/00335558008248231 CrossRefPubMedGoogle Scholar
  38. Posner, M. I., Snyder, C. R., & Davidson, B. J. (1980). Attention and the detection of signals. Journal of Experimental Psychology: General, 109, 160–174. doi:10.1037/0096-3445.109.2.160 CrossRefGoogle Scholar
  39. Robertson, L. C. (2004). Space, objects, minds, and brains. New York, NY: Psychology Press.Google Scholar
  40. Schnoebelen, T., & Kuperman, V. (2010). Using Amazon Mechanical Turk for linguistic research. Psihologija, 43, 441–464.CrossRefGoogle Scholar
  41. Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43, 155–167. doi:10.3758/s13428-010-0039-7 CrossRefPubMedGoogle Scholar
  42. Tootell, R. B. H., Switkes, E., Silverman, M. S., & Hamilton, S. L. (1988). Functional anatomy of macaque striate cortex. II. Retinotopic organization. Journal of Neuroscience, 8, 1531–1568.PubMedGoogle Scholar
  43. Van Essen, D. C., Newsome, W. T., & Maunsell, J. H. (1984). The visual field representation in striate cortex of the macaque monkey: Asymmetries, anisotropies, and individual variability. Vision Research, 24, 429–448.CrossRefPubMedGoogle Scholar
  44. Witzel, J., Cornelius, S., Witzel, N., Forster, K. I., & Forster, J. C. (2013). Testing the viability of webDMDX for masked priming experiments. Mental Lexicon, 8, 421–449.Google Scholar
  45. Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. PeerJ, 3(e1058), 1–35. doi:10.7717/peerj.1058 Google Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  • Joseph R. Pauszek
    • 1
  • Pedro Sztybel
    • 1
  • Bradley S. Gibson
    • 1
    • 2
  1. 1.University of Notre DameNotre DameUSA
  2. 2.Department of PsychologyUniversity of Notre DameNotre DameUSA

Personalised recommendations