Behavior Research Methods

, Volume 40, Issue 1, pp 322–327 | Cite as

The high-hurdle technique put to the test: Failure to find evidence that increasing loading times enhances data quality in Web-based studies



Two Web-based experiments examined the usefulness of artificially delaying the loading of the first page of the study. The idea pursued in this technique is to filter out less-motivated respondents through a higher respondent burden in the form of waiting time. Participants who remain in the study despite having had to wait for the first page of the study to appear on the screen are expected to be more highly motivated, and thus to produce data of higher quality. In both experiments, as expected, the longer the loading time, the lower the likelihood of people responding to the study. However, contrary to expectation, the dropout rate and quality of data were independent of the loading time. Therefore, artificially delaying the loading of the first page of the study is counterproductive.


Bayesian Information Crite Linear Decline Respondent Burden Online Panel Paranormal Belief 


  1. Agresti, A. (2002). Categorical data analysis. New York: Wiley.CrossRefGoogle Scholar
  2. Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832.CrossRefPubMedGoogle Scholar
  3. Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19, 146–162.CrossRefGoogle Scholar
  4. Göritz, A. S. (2004). The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels. International Journal of Market Research, 46, 327–345.Google Scholar
  5. Göritz, A. S. (2005). Contingent versus unconditional incentives in WWW-studies. Metodoloski Zvezki—Advances in Methodology & Statistics, 2, 1–14.Google Scholar
  6. Göritz, A. S. (2006). Incentives in Web studies: Methodological issues and a review. International Journal of Internet Science, 1, 58–70.Google Scholar
  7. Göritz, A. S. (2007). Using online panels in psychological research. In A. N. Joinson, K. Y. A. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 473–485). Oxford: Oxford University Press.Google Scholar
  8. Göritz, A. S., & Schumacher, J. (2000). The WWW as a research medium: An illustrative survey on paranormal belief. Perceptual & Motor Skills, 90, 1195–1206.CrossRefGoogle Scholar
  9. Heerwegh, D., & Loosveldt, G. (2002). Web surveys: The effect of controlling survey access using PIN numbers. Social Science Computer Review, 20, 10–21.CrossRefGoogle Scholar
  10. Heerwegh, D., & Loosveidt, G. (2003). An evaluation of the semiautomatic login procedure to control Web survey access. Social Science Computer Review, 21, 223–234.CrossRefGoogle Scholar
  11. Lazar, J., & Norcio, A. (1999). To err or not to err, that is the question: Novice user perception of errors while surfing the Web. In M. Khosrowpouv (Ed.), Proceedings of the Information Resource Management Association 1999 International Conference (pp. 321–325). Hershey, PA: Idea Group.Google Scholar
  12. Nielsen, J. (2000). Designing Web usability: The practice of simplicity. Indianapolis: New Riders.Google Scholar
  13. O’Neil, K. M., Penrod, S. D., & Bornstein, B. H. (2003). Web-based research: Methodological variables’ effects on dropout and sample characteristics. Behavior Research Methods, Instruments, & Computers, 35, 217–226.CrossRefGoogle Scholar
  14. Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89–117). San Diego: Academic Press.CrossRefGoogle Scholar
  15. Reips, U.-D. (2002a). Internet-based psychological experimenting: Five dos and five don’ts. Social Science Computer Review, 20, 241–249.Google Scholar
  16. Reips, U.-D. (2002b). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256.PubMedGoogle Scholar
  17. Reips, U.-D. (2002c). Theory and techniques of conducting Web experiments. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online social sciences (pp. 229–250). Seattle: Hogrefe & Huber.Google Scholar
  18. Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8, 448–467.CrossRefPubMedGoogle Scholar
  19. Schmidt, W. C. (2000). The server-side of psychology Web experiments. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 285–310). San Diego: Academic Press.CrossRefGoogle Scholar
  20. Schwarz, S., & Reips, U.-D. (2001). CGI versus JavaScript: A Web experiment on the reversed hindsight bias. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 75–90). Lengerich, Germany: Pabst Science.Google Scholar
  21. Sears, A., Jacko, J., & Dubach, E. (2000). International aspects of WWW usability and the role of high-end graphical enhancements. International Journal of Human-Computer Interaction, 12, 243–263.Google Scholar
  22. Shneiderman, B. (1998). Designing the user interface: Strategies for effective human-computer interaction (3rd ed.). Reading, MA: Addison-Wesley.Google Scholar
  23. Skttka, L. J., & Sargis, E. G. (2006). The Internet as psychological laboratory. Annual Review of Psychology, 57, 529–555.CrossRefGoogle Scholar
  24. Stieger, S., Reips, U.-D., & Voracek, M. (2007). Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout. Journal of the American Society for Information Science & Technology, 58, 1653–1660.CrossRefGoogle Scholar
  25. Vermunt, J. K. (1997a). LEM: A general program for the analysis of categorical data [Computer program]. Tilburg, The Netherlands: Tilburg University.Google Scholar
  26. Vermunt, J. K. (1997b). Log-linear models for event histories. Thousand Oaks, CA: Sage.Google Scholar
  27. Wenzel, O. (2001). Webdesign, Informationssuche und Flow: Nutzerverhalten auf unterschiedlich strukturierten Websites [Web design, search for information, and flow: User behavior on differently structured Web sites]. Lohmar, Germany: Eul.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2008

Authors and Affiliations

  1. 1.Organizational and Social PsychologyUniversity of Erlangen-NurembergNurembergGermany
  2. 2.Medical University of ViennaViennaAustria

Personalised recommendations