The state of web-based research: A survey and call for inclusion in curricula
The first papers that reported on conducting psychological research on the web were presented at the Society for Computers in Psychology conference 20 years ago, in 1996. Since that time, there has been an explosive increase in the number of studies that use the web for data collection. As such, it seems a good time, 20 years on, to examine the health and adoption of sound practices of research on the web. The number of studies conducted online has increased dramatically. Overall, it seems that the web can be a method for conducting valid psychological studies. However, it is less clear that students and researchers are aware of the nature of web research. While many studies are well conducted, there is also a certain laxness appearing regarding the design and conduct of online studies. This laxness appears both anecdotally to the authors as managers of large sites for posting links to online studies, and in a survey of current researchers. One of the deficiencies discovered is that there is no coherent approach to educating researchers as to the unique features of web research.
KeywordsWeb research Methodology Education Internet-based research Internet science Online research
The first papers to use the World-Wide Web as a research tool were presented at the 1996 Society for Computers in Psychology conference in Chicago. Most of these papers were published in the following May issue of Behavior Research Methods, Instruments & Computers, the name of the current journal until 2005. Some of the papers presented include papers examining how to do research on the web (e.g., Schmidt, 1997) and two papers that actually reported the results of experiments conducted on the web in 1995 (Krantz, Ballard, & Scher 1997; Reips, 1996, 1997). As such, web research has been conducted online for more than 20 years now (Reips, 2015) and it seems a good time to examine the growth and health of the use of the web as a research tool. (See also Wolfe (2017) in this issue for another perspective.)
A brief history
Before examining the current state of web research, it seems good to review how the field got to this point in a brief overview. When the first researchers were using the web, they had no guide other than curiosity about this method. As such, these first researchers were careful and exploratory. The primary focus of a number of the early studies was to determine if the web yielded reliable and valid results (e.g., Krantz, 1997; Krantz & Dalal, 2000) and if it could help in solving some of the issues that limited laboratory research, such as low power, limited external/ecological validity, and low generalizability (Reips, 1996, 1997, 2000). It is a bit of an understatement, given the large number of online studies conducted today, that many of the early findings were encouraging. Still, there were issues faced by these researchers that were thought to potentially impact the quality of the data. Some of the issues that have been studied were the high rates of drop-out of web participants, the possibility of data fraud, multiple submissions, differences in measurement, and the possibility that web samples differed from laboratory samples and – for some research purposes - from the population at large. These issues led some of these early researchers to examine the issues and if possible find ways to ameliorate them. Many researchers collected both laboratory and web samples to compare the results (e.g., Krantz, 1997 – and other examples). In this way, differences between web and laboratory samples could be observed. Sometimes the laboratory and web led to different results (see Krantz & Dalal, 2000 for a summary of some of these early studies), leading to what later became known as the (non-)equivalence debate (e.g., Buchanan, 2002, 2007). Ulf-Dietrich Reips (e.g., 1996, 1999) examined and developed techniques such as the high hurdle and warm-up techniques to reduce the degree of dropout during the study (Frick, Bächtiger, & Reips, 1999). He also pioneered the multiple-site entry technique to develop a way to determine if different ways of accessing the study or different sampling leads to different responses in studies (Reips, 2000, 2002). Researchers also examined different methods to test for data fraud (Schmidt, 2007).
More recently, there have been several articles that have examined the use of crowd-sourcing as a method of collecting data. The most common method has been to use Amazon’s Mechanical Turk as a method (e.g., Burmester, Kwang, & Gosling, 2011; Chandler, Mueller, & Paolacci, 2014). Participants are called workers as they are paid and might do any number of tasks, not just psychological studies. The idea is that these workers would be better motivated to complete the studies since they are paid, small sums usually, for completion. Many studies find that data quality is comparable to studies posted on the open web (Buhrmester et al., 2011) but there are issues of non-naïveté among participants as they often repeat similar types of studies (Chandler, et al., 2014) and because they organize in unions and use forums to communicate about the tasks. Our own work finds Turkers to produce lower quality data than participants from other online sources, e.g. in a personality test development task they had faster response times and increasingly more so, and out of 64 items with different means, Turkers scored more in the middle of the scale in 50 items (Reips, Buffardi, & Kuhlmann, 2011).
A picture of the current state
At the current time, there is vigorous use of the web for psychological research and related fields. Experienced researchers and students use the web frequently to conduct research. For example, Krantz (1996) posted links to over 500 studies last year. Many of the studies are being conducted by student researchers, both undergraduate and graduate.
However, there does not seem to be a coherent approach to educating these new researchers. Neither Birnbaum (2001) nor Fraley (2004) deal with methods of data communication that many servers do not support. Moreover, these books do not cover more recent methods for data communication like AJAX and JSON. In addition, a quick review of textbooks for undergraduate research methods classes find that none of them cover online methodologies (e.g., Lewandowski Jr., Ciarocco, & Strohmertz, 2016; Nester & Schutt, 2015). Several cover specialized methods such as qualitative methodologies, case studies, and single-subject designs, but no mention is made of doing research online, let alone the specialized techniques appropriate for doing research online. This lack of coverage stands in stark contrast to the number of undergraduates engaging in online research as part of their undergraduate education. Both authors can attest to posting a large number of links to undergraduate research studies on their sites (Krantz, 1996; Reips & Lengler, 2005). This use of online research methodology by undergraduates has also been mentioned by colleagues (e.g., Mangan, personal communication, 17 November, 2016) and is evident from many departments licensing commercial online software and the many invitations to teach workshops and summer schools the authors and other pioneers of Internet-based research received and keep receiving. It seems, given the frequency of posting of online studies by undergraduates, that it is more likely that students will encounter online research methods than these important but less common methods such as those commonly mentioned in undergraduate textbooks.
These problems could well be the result of the lack of education in online research methods mentioned above. However, as scholars, we are all aware of the problems of anecdotal evidence. Thus, a survey of current researchers using the web was conducted. The survey serves as an update of the survey by Musch and Reips (2000) conducted on early experimental researchers on the web. Many of the questions were copied over with some new questions added and a few modifications to deal with the changes since the original survey was conducted. While this survey asks some of the same questions as Gureckis et al. (2015), there are a few differences. The questions, while overlapping, are not the same. The present survey is more comparable to Musch and Reips and asks a broader array of questions. Moreover, the present survey examines those that have done online research while Gureckis et al. also included participants that had not conducted an online study. Where there is overlap, the similarity or differences will be noted.
In the Musch and Reips (2000) study there were 21 researchers, mostly faculty, who were most of the early experimenters on the web who ran true experiments. The participants were recruited electronically via online groups and pages, and personal e-mails. Participants in the current survey were 71 researchers recruited via online posting of the study on Krantz (1996) and personal e-mails to researchers. These participants had varying levels of education. Almost half (46.5%) of the participants had a doctorate, 25.4% had some post-baccalaureate education, 15.5% had a bachelor’s degree while the rest were either in college or had some other form of qualification.
Most of the questions for the current survey were taken from Musch and Reips (2000), but limited to a single page, whereas the original survey was on three pages. These questions were of varied format including Likert, multiple selection, and open ended. The original study focused on the use of web experiments, the current study opened the questions to consider any form of Internet-based research on psychological topics. Most of the original researchers focused on experiments, but survey methodologies have become the dominant form of online psychological study. The questions asked addressed issues of why responding researchers used the online methodologies, concerns about online methods, technology they used, and questions about their particular study. Some questions from the original study were not repeated here as the focus was on determining the early history of web experimenting as well as attitudes. Additional questions were added to the current survey asking the current participants about testing of their study and their knowledge of experimental methodology, online methodology, and knowledge of the literature of online research. The current survey is still available at: http://psych2.hanover.edu/research/SeniorProjects/2016/Survey/
The survey was administered online. The participant was first given an informed consent page. Clicking to continue to the survey was construed as consent to participate. No identifying information was collected about the participants at the time, nor was the survey hosted on any site that might independently collect such identifying information. The participant then proceeded to the survey and after submitting the data was directed to a debriefing page. As there was no deception, the debriefing merely reiterated the informed consent and gave contact information for the first author.
One of the first questions examined by Musch and Reips (2000) is a set of questions asking “How important were the following factors for your decision to conduct your research on the web?” The questions were asked on a 7-point Likert scale scored from 0 to 6 with the higher the rating the more important the reason. The answers ranged from “Not important at all” to “Very important.” The same questions were asked in the current study with an additional question to rate the ease of doing the study. Figure 3 shows the average and 95% confidence intervals for the ratings on these questions. The dark bars are from the Musch and Reips (2000) study and the lighter bars are from the current participants. Overall, it seems that similar factors were important to the early and current researchers. Number of participants and statistical power are very important to both the original and current participants. Large sample size also turned up as important to almost all of the respondents in Gureckis et al. (2015). It is interesting to note that this desire for larger samples sizes does not seem to translate to overall greater statistical power (Wolfe, 2017). One apparent difference in the present results and those of Gureckis et al. is the fact that almost all participants reported fast data collection as important, the most often reported benefit in that study, while speed is important but not the most important item in either the present survey or Musch and Reips (2000). The difference may lay in the way participants responded. In Gureckis, the number of participants that selected each option was recorded, and in the present study and that by Musch and Reips, the participants rank ordered the importance of speed of data collection. These combined studies suggest that speed is important to almost all researchers but perhaps rarely the most important criteria in doing a web study. The ability to replicate lab studies and reach special populations remains less important. The lack of interest in reaching special populations is somewhat perplexing as this is one of the unique abilities of online research to greatly extend the boundaries of psychological knowledge (Birnbaum, 1999; Mangan & Reips, 2007). The cost of the study is more important in the current sample. This factor was mentioned by about 75% of the respondents in Gureckis et al. (2015). This change may reflect the larger number of undergraduate students, the changes in ease of funding research, or a change in the population that does online research. A combination of reasons is also possible. The newer question about the ease of doing the study is also very highly rated. The current data suggests that cost and ease of study are the two most important factors in doing online research currently, though number of participants is nearly as highly rated.
The findings of the current survey indicate that, compared to Musch and Reips (2000), current researchers are more concerned about the low cost of doing an online study than before. The ease of doing a study is also highly rated. In addition, current participants trend toward being more concerned about fraud and ethical issues than original participants, but this increase is tempered by the fact that they still do not indicate great concern with either issue. Researchers seem to rely on previously validated materials for testing their studies and indicate a wide range of familiarity with the literature on web research.
On the positive side, many researchers do take the time to gain experience with web research in particular and the literature on doing web research. It is clear that some researchers are aware of the need to test their studies and validate their particular study.
However, there are several signs of concern. Beginning with the lack of presentation on web research in undergraduate textbooks there are a string of issues that suggest that many web researchers do not approach conducting an online study thinking about the unique issues raised by these online methods. First, researchers seem primarily motivated by number of participants, cost, and ease when choosing the web as a research platform (Fig. 3). It is particularly noteworthy that the ability to access special populations is not a highly ranked reason for doing web-based research. These choices suggest that the principal motivation for doing online research is convenience rather than considerations of whether this method is the best for getting the answers sought. When these observations are combined with the modest level of concern about issues of online research, the level of testing the study, and the fact that some researchers engage in online research with both little experience and little knowledge of web research methods, there is cause for concern that too much of the web research being conducted is being conducted inadequately.
It would be interesting to see what happens as more researchers use the emerging technologies to help them develop online studies (e.g., de Leeuw, 2015; Gureckis, et al., 2016; Lange, Kühn, & Filevich, 2015; Litman, Robinson, & Abberbock, 2016). Some of these methods are linked to crowd-sourcing, particularly Amazon Turk (Gureckis et al., 2016; Litman et al., 2016), but others seem designed to help research for studies using the web at large (de Leeuw, 2015; Lange et al., 2015). On the positive side, many of the best practices of web research can be incorporated into the online study methodology, which will reduce the need for the researchers to be informed about these best practices. For example, Reips designed WEXTOR (http://wextor.eu) from the beginning to automatically guide and nudge study authors into using best practices (e.g. non-obvious file naming) when creating web experiments with the tool. However, it seems unwise to completely rely on the study development platform to take care of all the pertinent design principles. A useful feature of these technologies would be tutorials and queries to help researches know what practices they ought to follow. For example, testing a study on multiple platforms is quite important. In experiments, this step can be vital (Krantz, 2001). A built-in query could ask if the study has been tested on different platforms when it is about to be published, much like you get queries when you try to delete a file. With so much of the literature available online these days, links to pertinent papers in the tool would be helpful as well. It still should be noted that these development tools do not help with one of the most persistent issues in all psychological research performed on computers, and that is the reliance on consumer grade equipment (Wolfe, 2017).
Our survey has its limitations. For example, as a self-report it cannot tell the difference between what is claimed and what is true about behavior and knowledge. While many researchers reported their materials had been validated, we do not know if they were validated for use on the Internet – and, in fact, many researchers may not be aware of the related literature that determined that an instrument needs to be tested in the mode it is later to be used in; online study materials need to be validated for online use (Buchanan, 2007; Buchanan, Johnson, & Goldberg, 2005).
Of course the rapid development of Internet-based research methods, including the use of mobile devices for tracking throughout a period of time (e.g., Stieger, Lewetz, & Reips, manuscript submitted for publication) and novel ways of using these devices, such as the accelerometer (Kuhlmann, Reips, & Stieger, 2017), to do non-reactive measurements could make textbook authors leery of adding web-based methods as they are in constant flux. However, most students will not be using the most advanced methods and the development of the apps for these studies is beyond the ability of most students. Most students will be conducting either experiments or surveys over the web and as such would benefit from basic instruction in web-based research methods. Perhaps the best response from the data here is for those faculties who have students who conduct research on the web to contact publishers of their research methods textbook to add content related to web-based research methods.
- Batinic, B., Reips, U.-D. & Bosnjak, M. (eds.)(2002). Online Social Sciences. Seattle: Hogrefe & Huber.Google Scholar
- Birnbaum, M. H. (ed.)(2000). Psychological Experiments on the Internet. New York: Academic Press.Google Scholar
- Birnbaum, M. H. (2001). Introduction to behavioral research on the Internet. Upper Saddle River: Prentice Hall.Google Scholar
- Buchanan, T. (2007). Personality testing on the Internet: What we know and what we do not. In A. N. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 447–459). Oxford: Oxford University Press.Google Scholar
- de Vaus, D. A. (1996). Surveys in social research. London: UCL Press.Google Scholar
- Fraley, R. C. (2004). How to conduct behavioral research over the Internet: A beginner’s guide to HTML and CGI/PERL. New York: The Guildford Press.Google Scholar
- Frick, A., Bächtiger, M. T., & Reips, U.-D. (1999). Financial incentives, personal information and drop-out in online studies. In U.-D. Reips, B. Batinic, W. Bandilla, M. Bosnjak, L. Gräf, K. Moser, & A. Werner (Eds.). Current Internet science - trends, techniques, results. Aktuelle Online-Forschung - Trends, Techniken, Ergebnisse. Zürich: Online Press. http://www.gor.de/gor99/tband99/pdfs/a_h/frick.pdf
- Gosling, S. D., & Johnson, J. A. (eds.)(2010). Advanced methods for conducting online behavioral research. Washington, DC, US: American Psychological Association.Google Scholar
- Joinson, A. N., McKenna, K., Postmes, T., & Reips, U.-D. (Eds.). (2007). The Oxford handbook of Internet psychology. Oxford: Oxford University Press.Google Scholar
- Krantz, J. H. (1996). Psychological Research on the Net. Available Online: http://psych.hanover.edu/research/exponnet.html
- Krantz, J. H. (2001). Stimulus delivery on the Web: What can be presented when calibration isn't possible. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 113–130). Lengerich: Pabst Science Publishers.Google Scholar
- Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness. Behavior Research Methods, Instruments, & Computers, 29, 264–269.Google Scholar
- Krantz, J. H., & Dalal, R. (2000). Validity of web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 35–60). New York: Academic Press.Google Scholar
- Lewandowski, G. W., Jr., Ciarocco, N. J., & Strohmetz, D. B. (2016). Discovering the scientist within: Research methods in psychology. New York: Macmillan.Google Scholar
- Musch, J., & Reips, U.-D. (2000). A brief history of Web experimenting. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 61–87). New York: Academic Press.Google Scholar
- Nestor, P. G., & Schutt, R. K. (2015). Research methods in psychology: Investigating human behavior (2nd ed.). Thousand Oak: Sage.Google Scholar
- Reips, U.-D. (1995). The Web Experimental Psychology Lab. Archived online and retrieved from: http://iscience.uni-konstanz.de/archive/ulf/Lab/WebExpPsyLab.html
- Reips, U.-D. (1996, October 31). Experimenting in the World Wide Web. Paper presented at the 26th Society for Computers in Psychology Conference (SCiP-96), Chicago. Retrieved from http://iscience.uni-konstanz.de/archive/reips/slides/scipchicago96.html
- Reips, U.-D. (1997). Das psychologische Experimentieren im Internet [Psychological experimenting on the Internet]. In B. Batinic (Ed.), Internet für Psychologen (pp. 245–265). Göttingen: Hogrefe.Google Scholar
- Reips, U.-D. (1999). Theorie und Techniken des Web-Experimentierens [Theory and techniques of Web experimenting]. In B. Batinic, A. Werner, L. Gräf, & W. Bandilla (Eds.), Online Research: Methoden, Anwendungen und Ergebnisse (pp. 277–296). Göttingen: Hogrefe.Google Scholar
- Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89–118). San Diego: Academic Press.Google Scholar
- Reips, U.-D. (2015). 20 años de revolución: La investigación científica con la ayuda de la Internet. Special issue "Dos décadas de Internet | Two decades of the public Internet". Infoamérica-ICR: Iberoamerican Communication Review, 9, 59–70.Google Scholar
- Reips, U.-D., & Bosnjak, M. (Eds.). (2001). Dimensions of Internet science. Lengerich: Pabst Science Publishers.Google Scholar
- Reips, U.-D., Buffardi, L., & Kuhlmann, T. (2011). Using Amazon’s Mechanical Turk for the recruitment of participants in Internet-based research. Paper presented at 13th General Online Research (GOR) conference. Germany: University of Düsseldorf.Google Scholar
- Reips, U.-D., & Musch, J. (Eds.). (2002). Internet-based psychological experimenting [Special issue]. Experimental Psychology, 49.Google Scholar
- Schmidt, W. C. (2007). Technical considerations when implementing online research. In A. N. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 461–472). Oxford: Oxford University Press.Google Scholar
- Taylor, J. (2002). Psychology and the Internet [Special issue]. Social Science Computer Review, 20 (3).Google Scholar
- Wolfe, C. R. (2017). Twenty years of Internet-based research at SCiP: A discussion of surviving concepts and new methodologies. Behavior Research Methods. Google Scholar