An Updated Protocol to Detect Invalid Entries in an Online Survey of Men Who Have Sex with Men (MSM): How Do Valid and Invalid Submissions Compare?
- 287 Downloads
Researchers use protocols to screen for suspicious survey submissions in online studies. We evaluated how well a de-duplication and cross-validation process detected invalid entries. Data were from the Sexually Explicit Media Study, an Internet-based HIV prevention survey of men who have sex with men. Using our protocol, 146 (11.6 %) of 1254 entries were identified as invalid. Most indicated changes to the screening questionnaire to gain entry (n = 109, 74.7 %), matched other submissions’ payment profiles (n = 56, 41.8 %), or featured an IP address that was recorded previously (n = 43, 29.5 %). We found few demographic or behavioral differences between valid and invalid samples, however. Invalid submissions had lower odds of reporting HIV testing in the past year (OR 0.63), and higher odds of requesting no payment compared to check payments (OR 2.75). Thus, rates of HIV testing would have been underestimated if invalid submissions had not been removed, and payment may not be the only incentive for invalid participation.
KeywordsSurvey methods Questionnaires Bias HIV Validity
The Sexually Explicit Media (SEM) Study was funded by the National Institute of Mental Health (NIMH), Grant #5R01MH087231-02. The authors are also grateful for the support of Barbara Lea in preparing this manuscript.
- 3.Rosser BRS, Konstan J, Gurak L, et al. Recruiting high risk Latino MSM for Internet prevention research. STD/HIV Prevention and the Internet Conference. Washington; 2001.Google Scholar
- 5.Bockting WO, Miner MH, Robinson BE, et al. Use of the Internet to reach the US transgender population for HIV/STD prevention research. STD/HIV Prevention and the Internet Conference. Washington; 2003.Google Scholar
- 12.Konstan JA, Rosser BRS, Ross MW, Stanton J, Edwards WM. The story of subject naught: a cautionary but optimistic tale of Internet survey research. J Comput Mediat Commun. 2005;10(2):11.Google Scholar
- 20.Konstan JA, Rosser BRS, Horvath KJ, Gurak L, Edwards W. Protecting subject data privacy in Internet-based HIV/STI prevention survey research. In: Conrad FG, Schober MF, editors. Envisioning the survey interview of the future. John Wiley & Sons: New York; 2008. ISBN: 978-0-471-78627-6.Google Scholar
- 22.Nosen E, Woody SR. Online surveys: effect of research design decision on rates of invalid participation and data credibility. Grad Stud J Psychol. 2008;10:3–14.Google Scholar
- 23.LimeSurvey Project Team, Schmitz C. LimeSurvey: an open source survey tool. LimeSurvey Project: Hamburg; 2011.Google Scholar
- 25.Rosser BRS. Online HIV prevention and the Men’s Internet Study-II (MINT-II): a glimpse into the world of designing highly-interactive sexual health Internet-based interventions. Canada: World Congress for Sexology. Montreal; 2005.Google Scholar
- 26.StataCorp. Stata Statistical Software: Release 12. StataCorp LP: College Station; 2011.Google Scholar