Skip to main content
SpringerLink
Log in

The Problem with Online Data Collection: Predicting Invalid Responding in Undergraduate Samples

  • Published:
Current Psychology Aims and scope Submit manuscript

Abstract

The popularity of online research is increasing, but the validity of the results obtained is not yet clear. The purpose of this study was to examine some of the factors that influence the validity of computerized data collection in an undergraduate sample. Participants were 99 university students randomly assigned to one of three data collection conditions: online survey platform, in-person computerized survey platform, and in-person pencil-and-paper survey. Results suggest self-reported inattention symptoms, exposure to more stressors, and computerized platforms predicted more invalid responding. In contrast, personality, self-reported impulsivity symptoms, and shorter completion times did not predict invalid responding. Overall, more than half of the participants failed at least one validity check and 11% failed three or more validity checks. Researchers, particularly those working with undergraduate samples, should consider implementing procedures to ensure the data collected online are valid.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Adlaf, E. M., Gliksman, L., Demers, A., & Newton-Taylor, B. (2001). The prevalence of elevated psychological distress among Canadian undergraduates: Findings from the 1998 Canadian campus survey. Journal of American College Health, 50, 67–72.

    Article  PubMed  Google Scholar 

  • Adler, L., & Cohen, J. (2004). Diagnosis and evaluation of adults with attention-deficit/hyperactivity disorder. The Psychiatric Clinics of North America, 27(2), 187–201. https://doi.org/10.1016/j.psc.2003.12.003.

    Article  PubMed  Google Scholar 

  • Alwin, D. F., & Krosnick, J. A. (1991). The reliability of survey attitude measurement: The influence of question and respondent attributes. Sociological Methods & Research, 20(1), 139–181. https://doi.org/10.1177/0049124191020001005.

    Article  Google Scholar 

  • Andrews, D., Nonnecke, B., & Preece, J. (2003). Electronic survey methodology: A case study in reaching hard-to-involve internet users. International Journal of Human-Computer Interaction, 16(2), 185–210.

    Article  Google Scholar 

  • Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45(2), 527–535. https://doi.org/10.3758/s13428-012-0265-2.

    Article  Google Scholar 

  • Balazs, J., & Kerestzeny, A. (2014). Subthreshold attention deficit hyperactivity in children and adolescents: A systematic review. European Child and Adolescent Psychiatry, 23, 393–408.

    Article  PubMed  Google Scholar 

  • Barratt, M. J., Ferris, J. A., & Lenton, S. (2015). Hidden populations, online purposive sampling, and external validity: Taking off the blindfold. Field Methods, 27(1), 3–21 https://doi.org/10.1177/1525822X14526838.

    Article  Google Scholar 

  • Bonini Campos, J. A. D., Zucoloto, M. L., Sampaio Bonafé, F. S., Jordani, P. C., & Maroco, J. (2011). Reliability and validity of self-reported burnout in college students: A cross randomized comparison of paper-and-pencil vs. online administration. Computers in Human Behavior, 27(5), 1875–1883 https://doi.org/10.1016/j.chb.2011.04.011.

    Article  Google Scholar 

  • Brock, R. L., Barry, R. A., Lawrence, E., Rolffs, J., Cerretani, J., & Zarling, A. (2015). Online administration of questionnaires assessing psychological, physical, and sexual aggression: Establishing psychometric equivalence. Psychology of Violence, 5, 294–304. https://doi.org/10.1037/a0037835.

    Article  Google Scholar 

  • Burnham, T. C., & Hare, B. (2007). Engineering human cooperation: Does involuntary neural activation increase public goods contributions? Human Nature, 18(2), 88–108. https://doi.org/10.1007/s12110-007-9012-2.

    Article  PubMed  Google Scholar 

  • Chetverikov, A., & Upravitelev, P. (2016). Online versus offline: The web as a medium for response time data collection. Behavior Research Methods, 48, 1086–1099. https://doi.org/10.3758/s13428-015-0632-x.

    Article  PubMed  Google Scholar 

  • Choi, S. H., Mitchell, J., & Lipkus, I. (2017). Lessons learned from an online study with dual-smoker couples. American Journal of Health Behavior, 41, 61–66. https://doi.org/10.5993/AJHB.41.1.6.

    Article  PubMed  Google Scholar 

  • Costa, R. R., & McCrae, P. T. (2003). Personality in adulthood : A five factor theory perspective. New York: Guilford Press.

    Google Scholar 

  • Culpepper, L. (2011). Recognizing and diagnosing ADHD in college students. Journal of Clinical Psychiatry, 72, e33. https://doi.org/10.4088/JCP.11009tx2c.

    Article  PubMed  Google Scholar 

  • Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006.

    Article  Google Scholar 

  • DeRight, J., & Jorgenen, R. S. (2015). I just want my research credit: Frequency of suboptimal effort in a non-clinical healthy undergraduate sample. The Clinical Neuropsychologist, 29, 101–117. https://doi.org/10.1080/13854046.2014.989267.

    Article  PubMed  Google Scholar 

  • Dillman, D. A. (2007). Mail and internet surveys: the tailored design method (2nd ed., 2007 update with new internet, visual, and mixed-mode guide). Hoboken: Wiley.

  • Döpfner, M., Steinhausen, H.-C., Coghill, D., Dalsgaard, S., Poole, L., Ralston, S. J., et al., ADORE Study Group. (2006). Cross-cultural reliability and validity of ADHD assessed by the ADHD rating scale in a pan-European study. European Child & Adolescent Psychiatry, 15(Suppl 1), I46–I55. https://doi.org/10.1007/s00787-006-1007-8.

  • Dumfart, B., & Neubauer, A. C. (2016). Conscientiousness is the most powerful noncognitive predictor of school achievement in adolescents. Journal of Individual Differences, 37, 8–15.

    Article  Google Scholar 

  • Epstein, J., & Klinkenberg, W. D. (2001). From Eliza to internet: A brief history of computerized assessment. Computers in Human Behavior, 17, 295–314.

    Article  Google Scholar 

  • Fielding, N., Lee, R., & Blank, G. (2008). The SAGE handbook of online research methods. Thousand Oaks: SAGE.

    Book  Google Scholar 

  • Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37, 504–528.

    Article  Google Scholar 

  • Grane, V. A., Endestad, T., Pinto, A. F., & Solbakk, A. K. (2014). Attentional control and subjective executive function in treatment-naive adults with attention deficit hyperactivity disorder. PloS One, 29, e115227. https://doi.org/10.1371/journal.pone.0115227.

    Article  Google Scholar 

  • Grey, J. A., Galos, D., Rosser, B. R., Konstan, J., Iantaffi, A., & Wilkerson, J. A. (2015). An updated protocol to detect invalid entries in an online survey of men who have sex with men (MSM): How do valid and invalid submissions compare? AIDS and Behavior, 19, 1928–1937. https://doi.org/10.1007/s10461-015-1033-y.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hardre, P. L., Crowson, H. M., & Xie, K. (2012). Examining contexts-of-use for web-based and paper-based questionnaires. Educational and Psychological Measurement, 72(6), 1015–1038. https://doi.org/10.1177/0013164412451977.

    Article  Google Scholar 

  • Houston, D. M., Sophie, K., & Allt, S. K. (1997). Psychological distress and error making among junior house officers. British Journal of Health Psychology, 2, 141–151.

    Article  Google Scholar 

  • Ihme, J. M., Lemke, F., Lieder, K., Martin, F., Müller, J. C., & Schmidt, S. (2009). Comparison of ability tests administered online and in the laboratory. Behavior Research Methods, 41(4), 1183–1189. https://doi.org/10.3758/BRM.41.4.1183.

    Article  PubMed  Google Scholar 

  • John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The big five inventory--versions 4a and 54. Berkeley: University of California, Berkeley, Institute of Personality and Social Research.

    Google Scholar 

  • Kim, H. S., & Hodgins, D. C. (2017). Reliability and validity of data obtained from alcohol, cannabis, and gambling populations on Amazon’s mechanical Turk. Psychology of. Addictive Behavior, 31, 85–94. https://doi.org/10.1037/adb0000219.

    Article  Google Scholar 

  • Kohn, P. M., Lafreniere, K., & Gurevich, M. (1990). The inventory of college students’ recent life experiences: A decontaminated hassles scale for a special population. Journal of Behavioral Medicine, 13(6), 619–630.

    Article  Google Scholar 

  • Lazarus, R. S. (1999). Stress and emotion: A new synthesis. New York: Springer.

    Google Scholar 

  • Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017). Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon’s mechanical Turk. Personality Disorders: Theory, Research, and Treatment, 8, 26–34. https://doi.org/10.1037/per0000191.

    Article  Google Scholar 

  • Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009.

    Article  Google Scholar 

  • Patalay, P., Hayes, D., Deighton, J., & Wolpert, M. (2016). A comparison of paper and computer administered strengths and difficulties questionnaire. Journal of Psychopathology and Behavioral Assessment, 38, 242–250. https://doi.org/10.1007/s10862-015-9507-9.

    Article  Google Scholar 

  • Ward, M. K., & Pond, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on internet-based surveys. Computers in Human Behavior, 48, 554–568. https://doi.org/10.1016/j.chb.2015.01.070.

    Article  Google Scholar 

Download references

Acknowledgements

The authors wish to thank Dr. Laszlo Erdodi for his comments on a previous version of this manuscript, and Dr. Kathryn Lafreniere and Ms. Joan Craig who made suggestions prior to the data collection. The authors also wish to thank Dragana Ostojic and Antonette Scavone who provided comments on this manuscript before it was submitted. This manuscript follows from the first author’s honours thesis work.

Funding

N/A

Author information

Authors and Affiliations

  1. Department of Psychology, University of Windsor, 401 Sunset Avenue, Windsor, ON N94, Canada

    Patricia Al-Salom & Carlin J. Miller

Authors
  1. Patricia Al-Salom
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlin J. Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlin J. Miller.

Ethics declarations

Conflict of Interest

Authors have no conflicts of interest to report.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committees and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Salom, P., Miller, C.J. The Problem with Online Data Collection: Predicting Invalid Responding in Undergraduate Samples. Curr Psychol 38, 1258–1264 (2019). https://doi.org/10.1007/s12144-017-9674-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12144-017-9674-9

Keywords

Search

Navigation