Behavior Research Methods

, Volume 49, Issue 5, pp 1621–1629 | Cite as

The state of web-based research: A survey and call for inclusion in curricula



The first papers that reported on conducting psychological research on the web were presented at the Society for Computers in Psychology conference 20 years ago, in 1996. Since that time, there has been an explosive increase in the number of studies that use the web for data collection. As such, it seems a good time, 20 years on, to examine the health and adoption of sound practices of research on the web. The number of studies conducted online has increased dramatically. Overall, it seems that the web can be a method for conducting valid psychological studies. However, it is less clear that students and researchers are aware of the nature of web research. While many studies are well conducted, there is also a certain laxness appearing regarding the design and conduct of online studies. This laxness appears both anecdotally to the authors as managers of large sites for posting links to online studies, and in a survey of current researchers. One of the deficiencies discovered is that there is no coherent approach to educating researchers as to the unique features of web research.


Web research Methodology Education Internet-based research Internet science Online research 


  1. Batinic, B., Reips, U.-D. & Bosnjak, M. (eds.)(2002). Online Social Sciences. Seattle: Hogrefe & Huber.Google Scholar
  2. Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399–407.CrossRefGoogle Scholar
  3. Birnbaum, M. H. (ed.)(2000). Psychological Experiments on the Internet. New York: Academic Press.Google Scholar
  4. Birnbaum, M. H. (2001). Introduction to behavioral research on the Internet. Upper Saddle River: Prentice Hall.Google Scholar
  5. Buchanan, T. (2002). Online assessment: Desirable or dangerous? Professional Psychology: Research and Practice, 33(2), 148–154.CrossRefGoogle Scholar
  6. Buchanan, T. (2007). Personality testing on the Internet: What we know and what we do not. In A. N. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 447–459). Oxford: Oxford University Press.Google Scholar
  7. Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21(2), 115–127.CrossRefGoogle Scholar
  8. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new sourse of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6(1), 3–5.CrossRefPubMedGoogle Scholar
  9. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.CrossRefPubMedGoogle Scholar
  10. de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47(1), 1–12.CrossRefPubMedGoogle Scholar
  11. de Vaus, D. A. (1996). Surveys in social research. London: UCL Press.Google Scholar
  12. Fraley, R. C. (2004). How to conduct behavioral research over the Internet: A beginner’s guide to HTML and CGI/PERL. New York: The Guildford Press.Google Scholar
  13. Frick, A., Bächtiger, M. T., & Reips, U.-D. (1999). Financial incentives, personal information and drop-out in online studies. In U.-D. Reips, B. Batinic, W. Bandilla, M. Bosnjak, L. Gräf, K. Moser, & A. Werner (Eds.). Current Internet science - trends, techniques, results. Aktuelle Online-Forschung - Trends, Techniken, Ergebnisse. Zürich: Online Press.
  14. Gosling, S. D., & Johnson, J. A. (eds.)(2010). Advanced methods for conducting online behavioral research. Washington, DC, US: American Psychological Association.Google Scholar
  15. Gureckis, T. M., Martin, J., McDonnell, J., Rich, A. S., Markant, D., Coenen, A., & Chan, P. (2016). psiTurk: An open-source framework for conducting replicable behavioral experiments online. Behavior Research Methods, 48(3), 829–842.CrossRefPubMedGoogle Scholar
  16. Joinson, A. N., McKenna, K., Postmes, T., & Reips, U.-D. (Eds.). (2007). The Oxford handbook of Internet psychology. Oxford: Oxford University Press.Google Scholar
  17. Krantz, J. H. (1996). Psychological Research on the Net. Available Online:
  18. Krantz, J. H. (2001). Stimulus delivery on the Web: What can be presented when calibration isn't possible. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 113–130). Lengerich: Pabst Science Publishers.Google Scholar
  19. Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness. Behavior Research Methods, Instruments, & Computers, 29, 264–269.Google Scholar
  20. Krantz, J. H., & Dalal, R. (2000). Validity of web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 35–60). New York: Academic Press.Google Scholar
  21. Lange, K., Kühn, S., & Filevich, E. (2015). "Just Another Tool for Online Studies” (JATOS): An easy solution for setup and management of web servers supporting online studies. PLoS ONE, 10(6), e0130834. doi: 10.1371/journal.pone.0130834 CrossRefPubMedPubMedCentralGoogle Scholar
  22. Lewandowski, G. W., Jr., Ciarocco, N. J., & Strohmetz, D. B. (2016). Discovering the scientist within: Research methods in psychology. New York: Macmillan.Google Scholar
  23. Litman, L., Robinson, J., & Abberbock, T. (2016). A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods. doi: 10.3758/s13428-016-0727-z PubMedCentralGoogle Scholar
  24. Mangan, M., & Reips, U.-D. (2007). Sleep, sex, and the Web: Surveying the difficult-to-reach clinical population suffering from sexsomnia. Behavior Research Methods, 39, 233–236.CrossRefPubMedGoogle Scholar
  25. Musch, J., & Reips, U.-D. (2000). A brief history of Web experimenting. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 61–87). New York: Academic Press.Google Scholar
  26. Nestor, P. G., & Schutt, R. K. (2015). Research methods in psychology: Investigating human behavior (2nd ed.). Thousand Oak: Sage.Google Scholar
  27. Reips, U.-D. (1995). The Web Experimental Psychology Lab. Archived online and retrieved from:
  28. Reips, U.-D. (1996, October 31). Experimenting in the World Wide Web. Paper presented at the 26th Society for Computers in Psychology Conference (SCiP-96), Chicago. Retrieved from
  29. Reips, U.-D. (1997). Das psychologische Experimentieren im Internet [Psychological experimenting on the Internet]. In B. Batinic (Ed.), Internet für Psychologen (pp. 245–265). Göttingen: Hogrefe.Google Scholar
  30. Reips, U.-D. (1999). Theorie und Techniken des Web-Experimentierens [Theory and techniques of Web experimenting]. In B. Batinic, A. Werner, L. Gräf, & W. Bandilla (Eds.), Online Research: Methoden, Anwendungen und Ergebnisse (pp. 277–296). Göttingen: Hogrefe.Google Scholar
  31. Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89–118). San Diego: Academic Press.Google Scholar
  32. Reips, U.-D. (2001). The Web Experimental Psychology Lab: Five years of data collection on the Internet. Behavior Research Methods, Instruments, & Computers, 33, 201–211.CrossRefGoogle Scholar
  33. Reips, U.-D. (2002). Internet-based psychological experimenting: Five dos and five don'ts. Social Science Computer Review, 20(3), 241–249. doi: 10.1177/089443930202000302 Google Scholar
  34. Reips, U.-D. (2010). Design and formatting in Internet-based research. In S. Gosling & J. Johnson (Eds.), Advanced methods for conducting online behavioral research (pp. 29–43). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  35. Reips, U.-D. (2015). 20 años de revolución: La investigación científica con la ayuda de la Internet. Special issue "Dos décadas de Internet | Two decades of the public Internet". Infoamérica-ICR: Iberoamerican Communication Review, 9, 59–70.Google Scholar
  36. Reips, U.-D., & Bosnjak, M. (Eds.). (2001). Dimensions of Internet science. Lengerich: Pabst Science Publishers.Google Scholar
  37. Reips, U.-D., Buffardi, L., & Kuhlmann, T. (2011). Using Amazon’s Mechanical Turk for the recruitment of participants in Internet-based research. Paper presented at 13th General Online Research (GOR) conference. Germany: University of Düsseldorf.Google Scholar
  38. Reips, U.-D., & Funke, F. (2008). Interval level measurement with visual analogue scales in Internet-based research: VAS Generator. Behavior Research Methods, 40, 699–704.CrossRefPubMedGoogle Scholar
  39. Reips, U.-D., & Lengler, R. (2005). The Web Experiment List: A Web service for the recruitment of participants and archiving of Internet-based experiments. Behavior Research Methods, 37, 287–292.CrossRefPubMedGoogle Scholar
  40. Reips, U.-D., & Musch, J. (Eds.). (2002). Internet-based psychological experimenting [Special issue]. Experimental Psychology, 49.Google Scholar
  41. Schmidt, W. C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments, & Computers, 29, 274–279.CrossRefGoogle Scholar
  42. Schmidt, W. C. (2007). Technical considerations when implementing online research. In A. N. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 461–472). Oxford: Oxford University Press.Google Scholar
  43. Taylor, J. (2002). Psychology and the Internet [Special issue]. Social Science Computer Review, 20 (3).Google Scholar
  44. Wolfe, C. R. (2017). Twenty years of Internet-based research at SCiP: A discussion of surviving concepts and new methodologies. Behavior Research Methods. Google Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  1. 1.Department of PsychologyHanover CollegeHanoverUSA
  2. 2.Department of PsychologyUniversity of KonstanzKonstanzGermany

Personalised recommendations