Abstract
Recruitment for HIV research among gay, bisexual, and other men who have sex with men (MSM) has increasingly moved to the online sphere. However, there are limited data comparing the characteristics of clinic-based respondents versus those recruited via online survey platforms. MSM were recruited from three sampling sites (STI clinic, MTurk, and Qualtrics) to participate in a survey from March 2015 to April 2016. Respondents were compared between each of the sampling sites on demographics, sexual history, substance use, and attention filter passage. Attention filter passage was high for the online sampling sites (MTurk = 93%; Qualtrics = 86%), but significantly lower for the clinic-based sampling site (72%). Clinic-based respondents were significantly more racially/ethnically diverse, reported lower income, and reported more unemployment than online respondents. Clinic-based respondents reported significantly more male sexual partners in the previous 3 months (M clinic-based = 6; MTurk = 3.6; Qualtrics = 4.5), a higher proportion of gonorrhea, chlamydia, and/or syphilis in the last year, and a greater proportion of methamphetamine use (clinic-based = 21%; MTurk = 5%), and inhaled nitrates use (clinic-based = 41%; MTurk = 11%). The clinic-based sample demonstrated more demographic diversity and a greater proportion of HIV risk behaviors when compared to the online samples, but also a relatively low attention filter passage rate. We recommend the use of attention filters across all modalities to assess response validity and urge caution with online survey engines as samples may differ demographically and behaviorally when compared to clinic-based respondents.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., & Zahs, D. (2010). Research synthesis. Public Opinion Quarterly, 74, 711–781. doi:10.1093/poq/nfq048.
Birnbaum, M. H. (2000). Psychological experiments on the internet. San Diego, CA: Academic.
Birnbaum, M. H. (2004). Human research and data collection via the internet. Annual Review of Psychology, 55, 803–832. doi:10.1146/annurev.psych.55.090902.141601.
Braunsberger, K., Wybenga, H., & Gates, R. (2007). A comparison of reliability between telephone and web-based surveys. Journal of Business Research, 60, 758–764. doi:10.1016/j.jbusres.2007.02.015.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980.
Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160. doi:10.1016/j.chb.2013.05.009.
Chang, L., & Krosnick, J. A. (2009). National surveys via RDD telephone interviewing versus the internet. Public Opinion Quarterly, 73, 641–678. doi:10.1093/poq/nfp075.
Grov, C. (2012). HIV risk and substance use in men who have sex with men surveyed in bathhouses, bars/clubs, and on Craigslist.org: Venue of recruitment matters. AIDS and Behavior, 16, 807–817. doi:10.1007/s10461-011-9999-6.
Grov, C., Bux, D., Jr., Parsons, J. T., & Morgenstern, J. (2009). Recruiting hard-to-reach drug-using men who have sex with men into an intervention study: Lessons learned and implications for applied research. Substance Use and Misuse, 44, 1855–1871. doi:10.3109/10826080802501570.
Grov, C., Cain, D., Whitfield, T. H. F., Rendina, H. J., Pawson, M., Ventuneac, A., & Parsons, J. T. (2016a). Recruiting a US national sample of HIV-negative gay and bisexual men to complete at-home self-administered HIV/STI testing and surveys: Challenges and opportunities. Sexuality Research and Social Policy, 13, 1–21. doi:10.1007/s13178-015-0212-y.
Grov, C., Rendina, H. J., Jimenez, R., & Parsons, J. T. (2016b). Using online settings to identify gay and bisexual men willing to take or with experience taking PrEP: Implications for researchers and providers. AIDS Education and Prevention, 28, 378–392.
Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400–407. doi:10.3758/s13428-015-0578-z.
Hernandez-Romieu, A. C., Sullivan, P. S., Sanchez, T. H., Kelley, C. F., Peterson, J. L., Del Rio, C., … Rosenberg, E. S. (2014). The comparability of men who have sex with men recruited from venue-time-space sampling and Facebook: A cohort study. JMIR Research Protocols, 3, e37. doi:10.2196/resprot.3342.
Holloway, I. W., Dunlap, S., Del Pino, H. E., Hermanstyne, K., Pulsipher, C., & Landovitz, R. J. (2014). Online social networking, sexual risk and protective behaviors: Considerations for clinicians and researchers. Current Addiction Reports, 1, 220–228. doi:10.1007/s40429-014-0029-4.
Marcus, U., Hickson, F., Weatherburn, P., Schmidt, A. J., & EMIS Network. (2013). Age biases in a large HIV and sexual behaviour-related internet survey among MSM. BMC Public Health. doi:10.1186/1471-2458-13-826.
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23. doi:10.3758/s13428-011-0124-6.
McKee, M. B., Picciano, J. F., Roffman, R. A., Swanson, F., & Kalichman, S. C. (2006). Marketing the “sex check”: Evaluating recruitment strategies for a telephone-based HIV prevention project for gay and bisexual men. AIDS Education and Prevention, 18, 116–131. doi:10.1521/aeap.2006.18.2.116.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi:10.1016/j.jesp.2009.03.009.
Parsons, J. T., Grov, C., & Golub, S. A. (2012). Sexual compulsivity, co-occurring psychosocial health problems, and HIV risk among gay and bisexual men: Further evidence of a syndemic. American Journal of Public Health, 102, 156–162. doi:10.2105/ajph.2011.300284.
Parsons, J. T., Grov, C., & Kelly, B. C. (2008). Comparing the effectiveness of two forms of time-space sampling to identify club drug-using young adults. Journal of Drug Issues, 38, 1061–1081.
Parsons, J. T., Vial, A. C., Starks, T. J., & Golub, S. A. (2013). Recruiting drug using men who have sex with men in behavioral intervention trials: A comparison of internet and field-based strategies. AIDS and Behavior, 17, 688–699. doi:10.1007/s10461-012-0231-0.
Paul, J. P., Catania, J., Pollack, L., & Stall, R. (2001). Understanding childhood sexual abuse as a predictor of sexual risk-taking among men who have sex with men: The Urban Men’s Health Study. Child Abuse and Neglect, 25, 557–584. doi:10.1016/s0145-2134(01)00226-5.
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y.
Rankin, K. M., Rauscher, G. H., McCarthy, B., Erdal, S., Lada, P., Il’yasova, D., & Davis, F. (2008). Comparing the reliability of responses to telephone-administered versus self-administered web-based surveys in a case-control study of adult malignant brain cancer. Cancer Epidemiology, Biomarkers and Prevention, 17, 2639–2646. doi:10.1158/1055-9965.epi-09-0304.
Raymond, H. F., Rebchook, G., Curotto, A., Vaudrey, J., Amsden, M., Levine, D., & McFarland, W. (2010). Comparing internet-based and venue-based methods to sample MSM in the San Francisco Bay Area. AIDS and Behavior, 14, 218–224. doi:10.1007/s10461-009-9521-6.
Rosser, B. R. S., Miner, M. H., Bockting, W. O., Ross, M. W., Konstan, J., Gurak, L., & Coleman, E. (2009). HIV risk and the internet: Results of the Men’s INTernet Sex (MINTS) Study. AIDS and Behavior, 13, 746–756. doi:10.1007/s10461-008-9399-8.
Simons, D. J., & Chabris, C. F. (2012). Common (mis)beliefs about memory: A replication and comparison of telephone and Mechanical Turk survey methods. PLoS ONE. doi:10.1371/journal.pone.0051876.
Sullivan, P. S., Khosropour, C. M., Luisi, N., Amsden, M., Coggia, T., Wingood, G. M., & DiClemente, R. J. (2011). Bias in online recruitment and retention of racial and ethnic minority men who have sex with men. Journal of Medical Internet Research. doi:10.2196/jmir.1797.
Touvier, M., Kesse-Guyot, E., Mejean, C., Pollet, C., Malon, A., Castetbon, K., & Hercberg, S. (2011). Comparison between an interactive web-based self-administered 24 h dietary record and an interview by a dietitian for large-scale epidemiological studies. British Journal of Nutrition, 105, 1055–1064. doi:10.1017/s0007114510004617.
Vial, A. C., Starks, T. J., & Parsons, J. T. (2014). Finding and recruiting the highest risk HIV-negative men who have sex with men. AIDS Education and Prevention, 26, 56–67. doi:10.1521/aeap.2014.26.1.56.
Vial, A. C., Starks, T. J., & Parsons, J. T. (2015). Relative efficiency of field and online strategies in the recruitment of HIV-positive men who have sex with men. AIDS Education and Prevention, 27, 103–111. doi:10.1521/aeap.2015.27.2.103.
Acknowledgements
The authors would like to acknowledge the work of the Los Angeles LGBT Center’s HIV/STI testing counselors for recruiting participants for the clinic-based arm of the study.
Funding
This work was funded by the UCLA School of Public Affairs and the Los Angeles LGBT Center. MRB was supported by the UCLA Postdoctoral Fellowship Training Program in Global HIV Prevention Research (Currier and Gorbach, PIs); T32MH080634. IWH was supported by California HIV/AIDS Research Program (CHRP; Grant RP15-LA-007); the Center for HIV Identification, Prevention, and Treatment (CHIPTS; NIMH Grant MH58107); the UCLA Center for AIDS Research (CFAR; NIAID Grant AI028697); and the National Center for Advancing Translational Sciences through UCLA (CSTI; Grant UL1TR000124). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors disclose no conflicts of interest.
Research involving Human Participants and/or Animals
This study was approved by the University of California, Los Angeles North General Institutional Review Board (IRB00004474; Project No. 14-000750).
Informed Consent
Informed consent was provided for each participant completing the survey. An example of the informed consent can be found here: https://uscsocialwork.qualtrics.com/SE/?SID=SV_beXsWwODysDi2od.
Appendix
Appendix
MTurk Recruitment Script
The survey is to assess the feasibility of conducting online research among gay and bisexual men. The survey will take approximately 15 min and ask various questions about your health and demographics.
Clinic-Based Recruitment Script
The Los Angeles LGBT Center is conducting a study in collaboration with UCLA to determine if online samples of gay, bisexual, and other men who have sex with men are comparable to individuals who come to The Los Angeles LGBT Center for HIV/STD screening on sexual health behaviors. The survey will last between 10 and 20 min, and you will be given a $10 gift card to either Target or Trader Joe’s for your participation. You are not required to participate, and your agreement or rejection to participate will NOT affect either current or future services at The Los Angeles LGBT Center. Are you interested in participating?
Rights and permissions
About this article
Cite this article
Beymer, M.R., Holloway, I.W. & Grov, C. Comparing Self-Reported Demographic and Sexual Behavioral Factors Among Men Who Have Sex with Men Recruited Through Mechanical Turk, Qualtrics, and a HIV/STI Clinic-Based Sample: Implications for Researchers and Providers. Arch Sex Behav 47, 133–142 (2018). https://doi.org/10.1007/s10508-016-0932-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10508-016-0932-y