AIDS and Behavior

, Volume 19, Issue 10, pp 1928–1937 | Cite as

An Updated Protocol to Detect Invalid Entries in an Online Survey of Men Who Have Sex with Men (MSM): How Do Valid and Invalid Submissions Compare?

  • Jeremy A. GreyEmail author
  • Joseph Konstan
  • Alex Iantaffi
  • J. Michael Wilkerson
  • Dylan Galos
  • B. R. Simon Rosser
Original Paper


Researchers use protocols to screen for suspicious survey submissions in online studies. We evaluated how well a de-duplication and cross-validation process detected invalid entries. Data were from the Sexually Explicit Media Study, an Internet-based HIV prevention survey of men who have sex with men. Using our protocol, 146 (11.6 %) of 1254 entries were identified as invalid. Most indicated changes to the screening questionnaire to gain entry (n = 109, 74.7 %), matched other submissions’ payment profiles (n = 56, 41.8 %), or featured an IP address that was recorded previously (n = 43, 29.5 %). We found few demographic or behavioral differences between valid and invalid samples, however. Invalid submissions had lower odds of reporting HIV testing in the past year (OR 0.63), and higher odds of requesting no payment compared to check payments (OR 2.75). Thus, rates of HIV testing would have been underestimated if invalid submissions had not been removed, and payment may not be the only incentive for invalid participation.


Survey methods Questionnaires Bias HIV Validity 



The Sexually Explicit Media (SEM) Study was funded by the National Institute of Mental Health (NIMH), Grant #5R01MH087231-02. The authors are also grateful for the support of Barbara Lea in preparing this manuscript.


  1. 1.
    van Gelder MMHJ, Bretveld RW, Roeleveld N. Web-based questionnaires: the future in epidemiology? Am J Epidemiol. 2010;172(11):1292.CrossRefPubMedGoogle Scholar
  2. 2.
    Bowen A. Internet sexuality research with rural men who have sex with men: can we recruit and retain them? J Sex Res. 2005;42(4):317–23.CrossRefPubMedGoogle Scholar
  3. 3.
    Rosser BRS, Konstan J, Gurak L, et al. Recruiting high risk Latino MSM for Internet prevention research. STD/HIV Prevention and the Internet Conference. Washington; 2001.Google Scholar
  4. 4.
    Salyers Bull S, Lloyd L, Rietmeijer C, McFarlane M. Recruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: The Smart Sex Quest Project. AIDS Care. 2004;16(8):931–43.CrossRefPubMedGoogle Scholar
  5. 5.
    Bockting WO, Miner MH, Robinson BE, et al. Use of the Internet to reach the US transgender population for HIV/STD prevention research. STD/HIV Prevention and the Internet Conference. Washington; 2003.Google Scholar
  6. 6.
    Rosser BRS. Capturing the social demographics of hidden sexual minorities: an Internet study of the transgender population in the United States. Sex Res Soc Policy. 2007;4(2):50–64. doi: 10.1525/srsp.2007.4.2.50.CrossRefGoogle Scholar
  7. 7.
    Duncan D. The general well-being of recreational drug users: a survey on the WWW. Int J Drug Policy. 2000;11(5):315–23.CrossRefPubMedGoogle Scholar
  8. 8.
    Nicholson T. A survey of adult recreational drug use via the World Wide Web: the Drugnet Study. J Psychoact Drug. 1999;31(4):415–22.CrossRefGoogle Scholar
  9. 9.
    Skitka LJ, Sargis EG. The Internet as psychological laboratory. Annu Rev Psychol. 2006;57:529–55.CrossRefPubMedGoogle Scholar
  10. 10.
    Bowen AM, Daniel CM, Williams ML, Baird GL. Identifying multiple submissions in Internet research: preserving data integrity. AIDS Behav. 2008;12(6):964–73.PubMedCentralCrossRefPubMedGoogle Scholar
  11. 11.
    Gosling SD, Vazire S, Srivastava S, John OP. Should we trust Web-based studies? a comparative analysis of six preconceptions about Internet questionnaires. Am Psychol. 2004;59(2):93–104.CrossRefPubMedGoogle Scholar
  12. 12.
    Konstan JA, Rosser BRS, Ross MW, Stanton J, Edwards WM. The story of subject naught: a cautionary but optimistic tale of Internet survey research. J Comput Mediat Commun. 2005;10(2):11.Google Scholar
  13. 13.
    Pequegnat W, Rosser BRS, Bowen AM, et al. Conducting Internet-based HIV/STD prevention survey research: considerations in design and evaluation. AIDS Behav. 2007;11(4):505–21.CrossRefPubMedGoogle Scholar
  14. 14.
    Birnbaum MH. Human research and data collection via the Internet. Annu Rev Psychol. 2004;55:803–32.CrossRefPubMedGoogle Scholar
  15. 15.
    Rosser BRS, Miner MH, Bockting WO, et al. HIV risk and the Internet: results of the Men’s Internet Study (MINTS). AIDS Behav. 2009;13(4):746–56.PubMedCentralCrossRefPubMedGoogle Scholar
  16. 16.
    Bauermeister JA, Pingel E, Zimmerman M, Couper M, Carballo-Diéguez A, Strecher VJ. Data quality in HIV/AIDS Web-based surveys: handling invalid and suspicious data. Field Methods. 2012;24(3):272–91.PubMedCentralCrossRefPubMedGoogle Scholar
  17. 17.
    Lundevall-Unger P, Tranvik T. IP addresses—just a number? Int J Law Inf Technol. 2011;19(1):53–73. doi: 10.1093/ijlit/eaq013.CrossRefGoogle Scholar
  18. 18.
    Chiasson MA, Parsons JT, Tesoriero JM, Carballo-Dieguez A, Hirshfield A, Remien RH. HIV behavioral research online. J Urban Health. 2006;83(1):73–85.PubMedCentralCrossRefPubMedGoogle Scholar
  19. 19.
    Nosek BA, Banaji MR, Greenwald AG. E-research: ethics, security, design, and control in psychological research on the Internet. J Soc Issue. 2002;58(1):161–76. doi: 10.1111/1540-4560.00254.CrossRefGoogle Scholar
  20. 20.
    Konstan JA, Rosser BRS, Horvath KJ, Gurak L, Edwards W. Protecting subject data privacy in Internet-based HIV/STI prevention survey research. In: Conrad FG, Schober MF, editors. Envisioning the survey interview of the future. John Wiley & Sons: New York; 2008. ISBN: 978-0-471-78627-6.Google Scholar
  21. 21.
    Im EO, Chee W. Issues in protection of human subjects in Internet research. Nurs Res. 2002;51(4):266.CrossRefPubMedGoogle Scholar
  22. 22.
    Nosen E, Woody SR. Online surveys: effect of research design decision on rates of invalid participation and data credibility. Grad Stud J Psychol. 2008;10:3–14.Google Scholar
  23. 23.
    LimeSurvey Project Team, Schmitz C. LimeSurvey: an open source survey tool. LimeSurvey Project: Hamburg; 2011.Google Scholar
  24. 24.
    Couper MP. Technology trends in survey data collection. Soc Sci Comput Rev. 2005;23(4):486–501. doi: 10.1177/0894439305278972.CrossRefGoogle Scholar
  25. 25.
    Rosser BRS. Online HIV prevention and the Men’s Internet Study-II (MINT-II): a glimpse into the world of designing highly-interactive sexual health Internet-based interventions. Canada: World Congress for Sexology. Montreal; 2005.Google Scholar
  26. 26.
    StataCorp. Stata Statistical Software: Release 12. StataCorp LP: College Station; 2011.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Jeremy A. Grey
    • 1
    Email author
  • Joseph Konstan
    • 2
  • Alex Iantaffi
    • 3
  • J. Michael Wilkerson
    • 4
  • Dylan Galos
    • 5
  • B. R. Simon Rosser
    • 5
  1. 1.Department of Epidemiology, Rollins School of Public HealthEmory UniversityAtlantaUSA
  2. 2.Department of Computer Science & EngineeringUniversity of MinnesotaMinneapolisUSA
  3. 3.Department of Family Medicine & Community HealthUniversity of Minnesota Medical SchoolMinneapolisUSA
  4. 4.Division of Health Promotion & Behavioral SciencesThe University of Texas Health Sciences Center (UTHealth) at HoustonHoustonUSA
  5. 5.Division of Epidemiology & Community HealthUniversity of Minnesota School of Public HealthMinneapolisUSA

Personalised recommendations