Abstract
Two Web-based experiments examined the usefulness of artificially delaying the loading of the first page of the study. The idea pursued in this technique is to filter out less-motivated respondents through a higher respondent burden in the form of waiting time. Participants who remain in the study despite having had to wait for the first page of the study to appear on the screen are expected to be more highly motivated, and thus to produce data of higher quality. In both experiments, as expected, the longer the loading time, the lower the likelihood of people responding to the study. However, contrary to expectation, the dropout rate and quality of data were independent of the loading time. Therefore, artificially delaying the loading of the first page of the study is counterproductive.
Article PDF
Similar content being viewed by others
References
Agresti, A. (2002). Categorical data analysis. New York: Wiley.
Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832.
Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19, 146–162.
Göritz, A. S. (2004). The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels. International Journal of Market Research, 46, 327–345.
Göritz, A. S. (2005). Contingent versus unconditional incentives in WWW-studies. Metodoloski Zvezki—Advances in Methodology & Statistics, 2, 1–14.
Göritz, A. S. (2006). Incentives in Web studies: Methodological issues and a review. International Journal of Internet Science, 1, 58–70.
Göritz, A. S. (2007). Using online panels in psychological research. In A. N. Joinson, K. Y. A. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 473–485). Oxford: Oxford University Press.
Göritz, A. S., & Schumacher, J. (2000). The WWW as a research medium: An illustrative survey on paranormal belief. Perceptual & Motor Skills, 90, 1195–1206.
Heerwegh, D., & Loosveldt, G. (2002). Web surveys: The effect of controlling survey access using PIN numbers. Social Science Computer Review, 20, 10–21.
Heerwegh, D., & Loosveidt, G. (2003). An evaluation of the semiautomatic login procedure to control Web survey access. Social Science Computer Review, 21, 223–234.
Lazar, J., & Norcio, A. (1999). To err or not to err, that is the question: Novice user perception of errors while surfing the Web. In M. Khosrowpouv (Ed.), Proceedings of the Information Resource Management Association 1999 International Conference (pp. 321–325). Hershey, PA: Idea Group.
Nielsen, J. (2000). Designing Web usability: The practice of simplicity. Indianapolis: New Riders.
O’Neil, K. M., Penrod, S. D., & Bornstein, B. H. (2003). Web-based research: Methodological variables’ effects on dropout and sample characteristics. Behavior Research Methods, Instruments, & Computers, 35, 217–226.
Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89–117). San Diego: Academic Press.
Reips, U.-D. (2002a). Internet-based psychological experimenting: Five dos and five don’ts. Social Science Computer Review, 20, 241–249.
Reips, U.-D. (2002b). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256.
Reips, U.-D. (2002c). Theory and techniques of conducting Web experiments. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online social sciences (pp. 229–250). Seattle: Hogrefe & Huber.
Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8, 448–467.
Schmidt, W. C. (2000). The server-side of psychology Web experiments. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 285–310). San Diego: Academic Press.
Schwarz, S., & Reips, U.-D. (2001). CGI versus JavaScript: A Web experiment on the reversed hindsight bias. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 75–90). Lengerich, Germany: Pabst Science.
Sears, A., Jacko, J., & Dubach, E. (2000). International aspects of WWW usability and the role of high-end graphical enhancements. International Journal of Human-Computer Interaction, 12, 243–263.
Shneiderman, B. (1998). Designing the user interface: Strategies for effective human-computer interaction (3rd ed.). Reading, MA: Addison-Wesley.
Skttka, L. J., & Sargis, E. G. (2006). The Internet as psychological laboratory. Annual Review of Psychology, 57, 529–555.
Stieger, S., Reips, U.-D., & Voracek, M. (2007). Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout. Journal of the American Society for Information Science & Technology, 58, 1653–1660.
Vermunt, J. K. (1997a). LEM: A general program for the analysis of categorical data [Computer program]. Tilburg, The Netherlands: Tilburg University.
Vermunt, J. K. (1997b). Log-linear models for event histories. Thousand Oaks, CA: Sage.
Wenzel, O. (2001). Webdesign, Informationssuche und Flow: Nutzerverhalten auf unterschiedlich strukturierten Websites [Web design, search for information, and flow: User behavior on differently structured Web sites]. Lohmar, Germany: Eul.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by a University of Erlangen-Nuremberg postdoctoral scholarship (HWP) to the first author.
Rights and permissions
About this article
Cite this article
Göritz, A.S., Stieger, S. The high-hurdle technique put to the test: Failure to find evidence that increasing loading times enhances data quality in Web-based studies. Behav Res 40, 322–327 (2008). https://doi.org/10.3758/BRM.40.1.322
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BRM.40.1.322