Abstract
Researchers use protocols to screen for suspicious survey submissions in online studies. We evaluated how well a de-duplication and cross-validation process detected invalid entries. Data were from the Sexually Explicit Media Study, an Internet-based HIV prevention survey of men who have sex with men. Using our protocol, 146 (11.6 %) of 1254 entries were identified as invalid. Most indicated changes to the screening questionnaire to gain entry (n = 109, 74.7 %), matched other submissions’ payment profiles (n = 56, 41.8 %), or featured an IP address that was recorded previously (n = 43, 29.5 %). We found few demographic or behavioral differences between valid and invalid samples, however. Invalid submissions had lower odds of reporting HIV testing in the past year (OR 0.63), and higher odds of requesting no payment compared to check payments (OR 2.75). Thus, rates of HIV testing would have been underestimated if invalid submissions had not been removed, and payment may not be the only incentive for invalid participation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
van Gelder MMHJ, Bretveld RW, Roeleveld N. Web-based questionnaires: the future in epidemiology? Am J Epidemiol. 2010;172(11):1292.
Bowen A. Internet sexuality research with rural men who have sex with men: can we recruit and retain them? J Sex Res. 2005;42(4):317–23.
Rosser BRS, Konstan J, Gurak L, et al. Recruiting high risk Latino MSM for Internet prevention research. STD/HIV Prevention and the Internet Conference. Washington; 2001.
Salyers Bull S, Lloyd L, Rietmeijer C, McFarlane M. Recruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: The Smart Sex Quest Project. AIDS Care. 2004;16(8):931–43.
Bockting WO, Miner MH, Robinson BE, et al. Use of the Internet to reach the US transgender population for HIV/STD prevention research. STD/HIV Prevention and the Internet Conference. Washington; 2003.
Rosser BRS. Capturing the social demographics of hidden sexual minorities: an Internet study of the transgender population in the United States. Sex Res Soc Policy. 2007;4(2):50–64. doi:10.1525/srsp.2007.4.2.50.
Duncan D. The general well-being of recreational drug users: a survey on the WWW. Int J Drug Policy. 2000;11(5):315–23.
Nicholson T. A survey of adult recreational drug use via the World Wide Web: the Drugnet Study. J Psychoact Drug. 1999;31(4):415–22.
Skitka LJ, Sargis EG. The Internet as psychological laboratory. Annu Rev Psychol. 2006;57:529–55.
Bowen AM, Daniel CM, Williams ML, Baird GL. Identifying multiple submissions in Internet research: preserving data integrity. AIDS Behav. 2008;12(6):964–73.
Gosling SD, Vazire S, Srivastava S, John OP. Should we trust Web-based studies? a comparative analysis of six preconceptions about Internet questionnaires. Am Psychol. 2004;59(2):93–104.
Konstan JA, Rosser BRS, Ross MW, Stanton J, Edwards WM. The story of subject naught: a cautionary but optimistic tale of Internet survey research. J Comput Mediat Commun. 2005;10(2):11.
Pequegnat W, Rosser BRS, Bowen AM, et al. Conducting Internet-based HIV/STD prevention survey research: considerations in design and evaluation. AIDS Behav. 2007;11(4):505–21.
Birnbaum MH. Human research and data collection via the Internet. Annu Rev Psychol. 2004;55:803–32.
Rosser BRS, Miner MH, Bockting WO, et al. HIV risk and the Internet: results of the Men’s Internet Study (MINTS). AIDS Behav. 2009;13(4):746–56.
Bauermeister JA, Pingel E, Zimmerman M, Couper M, Carballo-Diéguez A, Strecher VJ. Data quality in HIV/AIDS Web-based surveys: handling invalid and suspicious data. Field Methods. 2012;24(3):272–91.
Lundevall-Unger P, Tranvik T. IP addresses—just a number? Int J Law Inf Technol. 2011;19(1):53–73. doi:10.1093/ijlit/eaq013.
Chiasson MA, Parsons JT, Tesoriero JM, Carballo-Dieguez A, Hirshfield A, Remien RH. HIV behavioral research online. J Urban Health. 2006;83(1):73–85.
Nosek BA, Banaji MR, Greenwald AG. E-research: ethics, security, design, and control in psychological research on the Internet. J Soc Issue. 2002;58(1):161–76. doi:10.1111/1540-4560.00254.
Konstan JA, Rosser BRS, Horvath KJ, Gurak L, Edwards W. Protecting subject data privacy in Internet-based HIV/STI prevention survey research. In: Conrad FG, Schober MF, editors. Envisioning the survey interview of the future. John Wiley & Sons: New York; 2008. ISBN: 978-0-471-78627-6.
Im EO, Chee W. Issues in protection of human subjects in Internet research. Nurs Res. 2002;51(4):266.
Nosen E, Woody SR. Online surveys: effect of research design decision on rates of invalid participation and data credibility. Grad Stud J Psychol. 2008;10:3–14.
LimeSurvey Project Team, Schmitz C. LimeSurvey: an open source survey tool. LimeSurvey Project: Hamburg; 2011.
Couper MP. Technology trends in survey data collection. Soc Sci Comput Rev. 2005;23(4):486–501. doi:10.1177/0894439305278972.
Rosser BRS. Online HIV prevention and the Men’s Internet Study-II (MINT-II): a glimpse into the world of designing highly-interactive sexual health Internet-based interventions. Canada: World Congress for Sexology. Montreal; 2005.
StataCorp. Stata Statistical Software: Release 12. StataCorp LP: College Station; 2011.
Acknowledgments
The Sexually Explicit Media (SEM) Study was funded by the National Institute of Mental Health (NIMH), Grant #5R01MH087231-02. The authors are also grateful for the support of Barbara Lea in preparing this manuscript.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Grey, J.A., Konstan, J., Iantaffi, A. et al. An Updated Protocol to Detect Invalid Entries in an Online Survey of Men Who Have Sex with Men (MSM): How Do Valid and Invalid Submissions Compare?. AIDS Behav 19, 1928–1937 (2015). https://doi.org/10.1007/s10461-015-1033-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10461-015-1033-y